text:
googleplus-grab
=============

More information about the archiving project can be found on the ArchiveTeam wiki: [Google+](http://archiveteam.org/index.php?title=Google+)

Setup instructions
=========================

Be sure to replace `YOURNICKHERE` with the nickname that you want to be shown as, on the tracker. You don't need to register it, just pick a nickname you like.

In most of the below cases, there will be a web interface running at http://localhost:8001/. If you don't know or care what this is, you can just ignore it—otherwise, it gives you a fancy view of what's going on.

**If anything goes wrong while running the commands below, please scroll down to the bottom of this page. There's troubleshooting information there.**

Running with a warrior
-------------------------

Follow the [instructions on the ArchiveTeam wiki](http://archiveteam.org/index.php?title=Warrior) for installing the Warrior, and select the "Google+" project in the Warrior interface.

Running without a warrior
-------------------------
To run this outside the warrior, clone this repository, cd into its directory and run:

    pip install --upgrade seesaw
    ./get-wget-lua.sh

then start downloading with:

    run-pipeline pipeline.py --concurrent 2 YOURNICKHERE

For more options, run:

    run-pipeline --help

If you don't have root access and/or your version of pip is very old, you can replace "pip install --upgrade seesaw" with:

    wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py ; python get-pip.py --user ; ~/.local/bin/pip install --upgrade --user seesaw

so that pip and seesaw are installed in your home, then run

    ~/.local/bin/run-pipeline pipeline.py --concurrent 2 YOURNICKHERE

Running multiple instances on different IPs
-------------------------------------------

This feature requires seesaw version 0.0.16 or greater. Use `pip install --upgrade seesaw` to upgrade.

Use the `--context-value` argument to pass in `bind_address=123.4.5.6` (replace the IP address with your own).

Example of running 2 threads, no web interface, and Wget binding of IP address:

    run-pipeline pipeline.py --concurrent 2 YOURNICKHERE --disable-web-server --context-value bind_address=123.4.5.6

Distribution-specific setup
-------------------------
### For Debian/Ubuntu:

    adduser --system --group --shell /bin/bash archiveteam
    apt-get update && apt-get install -y git-core libgnutls-dev lua5.1 liblua5.1-0 liblua5.1-0-dev screen python-dev python-pip bzip2 zlib1g-dev flex autoconf
    pip install --upgrade seesaw
    su -c "cd /home/archiveteam; git clone https://github.com/ArchiveTeam/googleplus-grab.git; cd googleplus-grab; ./get-wget-lua.sh" archiveteam
    screen su -c "cd /home/archiveteam/googleplus-grab/; run-pipeline pipeline.py --concurrent 2 --address '127.0.0.1' YOURNICKHERE" archiveteam
    [... ctrl+A D to detach ...]

In __Debian Jessie, Ubuntu 18.04 Bionic and above__, the `libgnutls-dev` package was renamed to `libgnutls28-dev`. So, you need to do the following instead:

    adduser --system --group --shell /bin/bash archiveteam
    apt-get update && apt-get install -y git-core libgnutls28-dev lua5.1 liblua5.1-0 liblua5.1-0-dev screen python-dev python-pip bzip2 zlib1g-dev flex autoconf
    [... pretty much the same as above ...]

Wget-lua is also available on [ArchiveTeam's PPA](https://launchpad.net/~archiveteam/+archive/wget-lua) for Ubuntu.

And you may also need to update your Python's warcio if you find screen detaching immediately after running the job in Ubuntu 18.04 Bionic and above:

    pip install warcio --upgrade

### For CentOS:

Ensure that you have the CentOS equivalent of bzip2 installed as well. You will the EPEL repository to be enabled.

    yum -y install autoconf automake flex gnutls-devel lua-devel python-pip zlib-devel
    pip install --upgrade seesaw
    [... pretty much the same as above ...]

### For openSUSE:

    zypper install liblua5_1 lua51 lua51-devel screen python-pip libgnutls-devel bzip2 python-devel gcc make
    pip install --upgrade seesaw
    [... pretty much the same as above ...]

### For OS X:

You need Homebrew. Ensure that you have the OS X equivalent of bzip2 installed as well.

    brew install python lua gnutls
    pip install --upgrade seesaw
    [... pretty much the same as above ...]

**There is a known issue with some packaged versions of rsync. If you get errors during the upload stage, googleplus-grab will not work with your rsync version.**

This supposedly fixes it:

    alias rsync=/usr/local/bin/rsync

### For Arch Linux:

Ensure that you have the Arch equivalent of bzip2 installed as well.

1. Make sure you have `python2-pip` installed.
2. Install [the wget-lua package from the AUR](https://aur.archlinux.org/packages/wget-lua/).
3. Run `pip2 install --upgrade seesaw`.
4. Modify the run-pipeline script in seesaw to point at `#!/usr/bin/python2` instead of `#!/usr/bin/python`.
5. `useradd --system --group users --shell /bin/bash --create-home archiveteam`
6. `screen su -c "cd /home/archiveteam/googleplus-grab/; run-pipeline pipeline.py --concurrent 2 --address '127.0.0.1' YOURNICKHERE" archiveteam`

### For Alpine Linux:

    apk add lua5.1 git python bzip2 bash rsync gcc libc-dev lua5.1-dev zlib-dev gnutls-dev autoconf flex make
    python -m ensurepip
    pip install -U seesaw
    git clone https://github.com/ArchiveTeam/googleplus-grab
    cd googleplus-grab; ./get-wget-lua.sh
    run-pipeline pipeline.py --concurrent 2 --address '127.0.0.1' YOURNICKHERE

### For FreeBSD:

Honestly, I have no idea. `./get-wget-lua.sh` supposedly doesn't work due to differences in the `tar` that ships with FreeBSD. Another problem is the apparent absence of Lua 5.1 development headers. If you figure this out, please do let us know on IRC (irc.efnet.org #archiveteam).

Troubleshooting
=========================

Broken? These are some of the possible solutions:

### wget-lua was not successfully built

If you get errors about `wget.pod` or something similar, the documentation failed to compile - wget-lua, however, compiled fine. Try this:

    cd get-wget-lua.tmp
    mv src/wget ../wget-lua
    cd ..

The `get-wget-lua.tmp` name may be inaccurate. If you have a folder with a similar but different name, use that instead and please let us know on IRC what folder name you had!

Optionally, if you know what you're doing, you may want to use wgetpod.patch.

### Problem with gnutls or openssl during get-wget-lua

Please ensure that gnutls-dev(el) and openssl-dev(el) are installed.

### ImportError: No module named seesaw

If you're sure that you followed the steps to install `seesaw`, permissions on your module directory may be set incorrectly. Try the following:

    chmod o+rX -R /usr/local/lib/python2.7/dist-packages

### run-pipeline: command not found

Install `seesaw` using `pip2` instead of `pip`.

    pip2 install seesaw

### Issues in the code

If you notice a bug and want to file a bug report, please use the GitHub issues tracker.

Are you a developer? Help write code for us! Look at our [developer documentation](http://archiveteam.org/index.php?title=Dev) for details.

### Other problems

Have an issue not listed here? Join us on IRC and ask! We can be found at irc.efnet.org #googleminus.


text:
import class AppKit.NSBackgroundActivityScheduler
import var AppKit.NSApp
import PMKFoundation
import Foundation
import PromiseKit
import Version
import Path

public class AppUpdater {
    var active = Promise()
#if !DEBUG
    let activity: NSBackgroundActivityScheduler
#endif
    let owner: String
    let repo: String

    var slug: String {
        return "\(owner)/\(repo)"
    }

    public var allowPrereleases = false

    public init(owner: String, repo: String) {
        self.owner = owner
        self.repo = repo
    #if DEBUG
        check().cauterize()
    #else
        activity = NSBackgroundActivityScheduler(identifier: "dev.mxcl.AppUpdater")
        activity.repeats = true
        activity.interval = 24 * 60 * 60
        activity.schedule { [unowned self] completion in
            guard !self.activity.shouldDefer, self.active.isResolved else {
                return completion(.deferred)
            }
            self.check().cauterize().finally {
                completion(.finished)
            }
        }
    #endif
    }

#if !DEBUG
    deinit {
        activity.invalidate()
    }
#endif

    private enum Error: Swift.Error {
        case bundleExecutableURL
        case codeSigningIdentity
        case invalidDownloadedBundle
    }

    public func check() -> Promise<Void> {
        guard active.isResolved else {
            return active
        }
        guard Bundle.main.executableURL != nil else {
            return Promise(error: Error.bundleExecutableURL)
        }
        let currentVersion = Bundle.main.version

        func validate(codeSigning b1: Bundle, _ b2: Bundle) -> Promise<Void> {
            return firstly {
                when(fulfilled: b1.codeSigningIdentity, b2.codeSigningIdentity)
            }.done {
                guard $0 == $1 else { throw Error.codeSigningIdentity }
            }
        }

        func update(with asset: Release.Asset) throws -> Promise<Void> {
        #if DEBUG
            print("notice: AppUpdater dry-run:", asset)
            return Promise()
        #else
            let tmpdir = try FileManager.default.url(for: .itemReplacementDirectory, in: .userDomainMask, appropriateFor: Bundle.main.bundleURL, create: true)

            return firstly {
                URLSession.shared.downloadTask(.promise, with: asset.browser_download_url, to: tmpdir.appendingPathComponent("download"))
            }.then { dst, _ in
                unzip(dst, contentType: asset.content_type)
            }.compactMap { downloadedAppBundle in
                Bundle(url: downloadedAppBundle)
            }.then { downloadedAppBundle in
                validate(codeSigning: .main, downloadedAppBundle).map{ downloadedAppBundle }
            }.done { downloadedAppBundle in

                // UNIX is cool. Delete ourselves, move new one in then restart.

                let installedAppBundle = Bundle.main
                guard let exe = downloadedAppBundle.executable, exe.exists else {
                    throw Error.invalidDownloadedBundle
                }
                let finalExecutable = installedAppBundle.path/exe.relative(to: downloadedAppBundle.path)

                try installedAppBundle.path.delete()
                try downloadedAppBundle.path.move(to: installedAppBundle.path)
                try FileManager.default.removeItem(at: tmpdir)

                let proc = Process()
                if #available(OSX 10.13, *) {
                    proc.executableURL = finalExecutable.url
                } else {
                    proc.launchPath = finalExecutable.string
                }
                proc.launch()

                // seems to work, though for sure, seems asking a lot for it to be reliable!
                //TODO be reliable! Probably get an external applescript to ask us this one to quit then exec the new one
                NSApp.terminate(self)
            }.ensure {
                _ = try? FileManager.default.removeItem(at: tmpdir)
            }
        #endif
        }

        let url = URL(string: "https://api.github.com/repos/\(slug)/releases")!

        active = firstly {
            URLSession.shared.dataTask(.promise, with: url).validate()
        }.map {
            try JSONDecoder().decode([Release].self, from: $0.data)
        }.compactMap { releases in
            try releases.findViableUpdate(appVersion: currentVersion, repo: self.repo, prerelease: self.allowPrereleases)
        }.then { asset in
            try update(with: asset)
        }

        return active
    }
}

private struct Release: Decodable {
    let tag_name: Version
    let prerelease: Bool
    struct Asset: Decodable {
        let name: String
        let browser_download_url: URL
        let content_type: ContentType
    }
    let assets: [Asset]

    func viableAsset(forRepo repo: String) -> Asset? {
        return assets.first(where: { (asset) -> Bool in
            let prefix = "\(repo.lowercased())-\(tag_name)"
            let name = (asset.name as NSString).deletingPathExtension.lowercased()

            switch (name, asset.content_type) {
            case ("\(prefix).tar", .tar):
                return true
            case (prefix, _):
                return true
            default:
                return false
            }
        })
    }
}

private enum ContentType: Decodable {
    init(from decoder: Decoder) throws {
        switch try decoder.singleValueContainer().decode(String.self) {
        case "application/x-bzip2", "application/x-xz", "application/x-gzip":
            self = .tar
        case "application/zip":
            self = .zip
        default:
            throw PMKError.badInput
        }
    }

    case zip
    case tar
}

extension Release: Comparable {
    static func < (lhs: Release, rhs: Release) -> Bool {
        return lhs.tag_name < rhs.tag_name
    }

    static func == (lhs: Release, rhs: Release) -> Bool {
        return lhs.tag_name == rhs.tag_name
    }
}

private extension Array where Element == Release {
    func findViableUpdate(appVersion: Version, repo: String, prerelease: Bool) throws -> Release.Asset? {
        let suitableReleases = prerelease ? self : filter{ !$0.prerelease }
        guard let latestRelease = suitableReleases.sorted().last else { return nil }
        guard appVersion < latestRelease.tag_name else { throw PMKError.cancelled }
        return latestRelease.viableAsset(forRepo: repo)
    }
}

private func unzip(_ url: URL, contentType: ContentType) -> Promise<URL> {

    let proc = Process()
    if #available(OSX 10.13, *) {
        proc.currentDirectoryURL = url.deletingLastPathComponent()
    } else {
        proc.currentDirectoryPath = url.deletingLastPathComponent().path
    }

    switch contentType {
    case .tar:
        proc.launchPath = "/usr/bin/tar"
        proc.arguments = ["xf", url.path]
    case .zip:
        proc.launchPath = "/usr/bin/unzip"
        proc.arguments = [url.path]
    }

    func findApp() throws -> URL? {
        let cnts = try FileManager.default.contentsOfDirectory(at: url.deletingLastPathComponent(), includingPropertiesForKeys: [.isDirectoryKey], options: .skipsSubdirectoryDescendants)
        for url in cnts {
            guard url.pathExtension == "app" else { continue }
            guard let foo = try url.resourceValues(forKeys: [.isDirectoryKey]).isDirectory, foo else { continue }
            return url
        }
        return nil
    }

    return firstly {
        proc.launch(.promise)
    }.compactMap { _ in
        try findApp()
    }
}

private extension Bundle {
    var isCodeSigned: Guarantee<Bool> {
        let proc = Process()
        proc.launchPath = "/usr/bin/codesign"
        proc.arguments = ["-dv", bundlePath]
        return proc.launch(.promise).map { _ in
            true
        }.recover { _ in
            .value(false)
        }
    }

    var codeSigningIdentity: Promise<String> {
        let proc = Process()
        proc.launchPath = "/usr/bin/codesign"
        proc.arguments = ["-dvvv", bundlePath]

        return firstly {
            proc.launch(.promise)
        }.compactMap {
            String(data: $0.err.fileHandleForReading.readDataToEndOfFile(), encoding: .utf8)
        }.map {
            $0.split(separator: "\n")
        }.filterValues {
            $0.hasPrefix("Authority=")
        }.firstValue.map { line in
            String(line.dropFirst(10))
        }
    }
}

repo_name: mxcl/AppUpdater

repo_language: Swift

mime_type: text/plain


text:
// swift-tools-version:4.2
import PackageDescription

let package = Package(
    name: "AppUpdater",
    products: [
        .library(name: "AppUpdater", targets: ["AppUpdater"]),
    ],
    dependencies: [
        .package(url: "https://github.com/PromiseKit/Foundation", from: "3.3.0"),
        .package(url: "https://github.com/mxcl/Path.swift", from: "1.0.0"),
        .package(url: "https://github.com/mxcl/Version", from: "1.0.0"),
    ],
    targets: [
        .target(name: "AppUpdater", dependencies: ["PMKFoundation", "Path", "Version"], path: ".", sources: ["AppUpdater.swift"]),
    ],
    swiftLanguageVersions: [.v4_2, .version("5")]
)


text:
# AppUpdater

A simple app-updater for macOS, checks your GitHub releases for a binary asset
once a day and silently updates your app.

## Caveats

* We make no allowances for ensuring your app is not being actively used by the user
    at the time of update. PR welcome.
* Assets must be named: `\(reponame)-\(semanticVersion).ext`.
* Will not work if App is installed as a root user.

## Features

* Full semantic versioning support: we understand alpha/beta etc.
* We check the code-sign identity of the download matches the app that is
    running before doing the update. Thus if you don’t code-sign I’m not sure what
    would happen.
* We support zip files or tarballs.

# Support mxcl

Hey there, I’m Max Howell. I’m a prolific producer of open source software and
probably you already use some of it (for example, I created [`brew`]). I work
full-time on open source and it’s hard; currently *I earn less than minimum
wage*. Please help me continue my work, I appreciate it 🙏🏻

[Other ways to say thanks](http://mxcl.dev/#donate).

[`brew`]: https://brew.sh

## Usage

```swift
package.dependencies.append(.package(url: "https://github.com/mxcl/AppUpdater.git", from: "1.0.0"))
```

Then:

```swift
import AppUpdater

@NSApplicationMain
class AppDelegate: NSObject, NSApplicationDelegate {

    let updater = AppUpdater(owner: "your-github-username", repo: "your-github-repo-name")

    //NOTE this is optional, the `AppUpdater` object schedules a daily update check itself
    @IBAction func userRequestedAnExplicitUpdateCheck() {
        updater.check().catch(policy: .allErrors) { error in
            if error.isCancelled {
                // promise is cancelled if we are already up-to-date
            } else {
                // show alert for this error
            }
        }
    }
}
```

## Alternatives

* [Sparkle](https://github.com/sparkle-project/Sparkle)
* [Squirrel](https://github.com/Squirrel/Squirrel.Mac)


text:
on:
  pull_request:
    paths:
      - '*.swift'
      - .github/workflows/ci.yml
    schedule:
      - cron: '3 3 * * 3'  # 3:03 AM, every Wednesday
jobs:
  build:
    runs-on: macos-10.15
    strategy:
      matrix:
        xcode: [^10, ^11, ^12]
        conf: [release, debug]
    steps:
    - uses: actions/checkout@v2
    - uses: mxcl/xcodebuild@v1
      with:
        xcode: ${{ matrix.xcode }}
        configuration: ${{ matrix.conf }}
        warnings-as-errors: true
        action: build


text:
!IF [where /Q Makefile.auto.win]
# The file doesn't exist, so don't include it.
!ELSE
!INCLUDE Makefile.auto.win
!ENDIF

!IF [cmd /C IF NOT EXIST deps\hoedown EXIT 1]
HOEDOWN_PATH = ..\hoedown
!ELSE
HOEDOWN_PATH = deps\hoedown
!ENDIF

NMAKE = nmake /$(MAKEFLAGS)
MIX = mix
CFLAGS = /O2 /EHsc /I"$(HOEDOWN_PATH)\src"

all: markdown

markdown:
	$(MIX) compile

Makefile.auto.win:
	echo # Auto-generated as part of Makefile.win, do not modify. > $@
	erl -eval "io:format(\"~s~n\", [lists:concat([\"ERTS_INCLUDE_PATH=\", code:root_dir(), \"/erts-\", erlang:system_info(version), \"/include\"])])" -s init stop -noshell >> $@

HOEDOWN_SRC = \
	$(HOEDOWN_PATH)\src\autolink.obj \
	$(HOEDOWN_PATH)\src\buffer.obj \
	$(HOEDOWN_PATH)\src\document.obj \
	$(HOEDOWN_PATH)\src\escape.obj \
	$(HOEDOWN_PATH)\src\html.obj \
	$(HOEDOWN_PATH)\src\html_blocks.obj \
	$(HOEDOWN_PATH)\src\html_smartypants.obj \
	$(HOEDOWN_PATH)\src\stack.obj \
	$(HOEDOWN_PATH)\src\version.obj

hoedown.dll:
	cd $(HOEDOWN_PATH)
	$(NMAKE) /F Makefile.win hoedown.dll
        cd $(MAKEDIR)

!IFDEF ERTS_INCLUDE_PATH
priv\markdown.dll: hoedown.dll
	$(CC) $(CFLAGS) /I"$(ERTS_INCLUDE_PATH)" /LD /MD /Fe$@ $(HOEDOWN_SRC) src\markdown.c
!ELSE
priv\markdown.dll: Makefile.auto.win
	$(NMAKE) /F Makefile.win priv\markdown.dll
!ENDIF

.c.obj:
	$(CC) $(CFLAGS) /c $< /Fo$@

.IGNORE:

clean:
	$(MIX) clean
	cd $(HOEDOWN_PATH)
	$(NMAKE) /F Makefile.win clean
	cd $(MAKEDIR)
	del /Q /F priv\markdown.dll priv\markdown.exp priv\markdown.lib markdown.obj
	del /Q /F Makefile.auto.win

repo_name: devinus/markdown

repo_language: C


text:
defmodule Mix.Tasks.Compile.Hoedown do
  def run(_) do
    if match? {:win32, _}, :os.type do
      {result, _error_code} = System.cmd("nmake", ["/F", "Makefile.win", "priv\\markdown.dll"], stderr_to_stdout: true)
      IO.binwrite result
    else
      {result, _error_code} = System.cmd("make", ["priv/markdown.so"], stderr_to_stdout: true)
      IO.binwrite result
    end
    :ok
  end
end

defmodule Markdown.Mixfile do
  use Mix.Project

  @version "0.1.0"

  def project do
    [app: :markdown,
     version: @version,
     elixir: ">= 0.14.3 and < 2.0.0",
     compilers: [:hoedown, :elixir, :app],
     deps: deps()]
  end

  def application do
    []
  end

  defp deps do
    [{:hoedown, github: "hoedown/hoedown", app: false}]
  end
end


text:
MIX = mix
CFLAGS = -g -O3 -ansi -pedantic -Wall -Wextra -Wno-unused-parameter

ERLANG_PATH = $(shell erl -eval 'io:format("~s", [lists:concat([code:root_dir(), "/erts-", erlang:system_info(version), "/include"])])' -s init stop -noshell)
CFLAGS += -I$(ERLANG_PATH)

ifeq ($(wildcard deps/hoedown),)
	HOEDOWN_PATH = ../hoedown
else
	HOEDOWN_PATH = deps/hoedown
endif

CFLAGS += -I$(HOEDOWN_PATH)/src

ifneq ($(OS),Windows_NT)
	CFLAGS += -fPIC

	ifeq ($(shell uname),Darwin)
		LDFLAGS += -dynamiclib -undefined dynamic_lookup
	endif
endif

.PHONY: all markdown clean

all: markdown

markdown:
	$(MIX) compile

priv/markdown.so: src/markdown.c
	$(MAKE) -C $(HOEDOWN_PATH) libhoedown.a
	$(CC) $(CFLAGS) -shared $(LDFLAGS) -o $@ src/markdown.c $(HOEDOWN_PATH)/libhoedown.a

clean:
	$(MIX) clean
	$(MAKE) -C $(HOEDOWN_PATH) clean
	$(RM) priv/markdown.so


text:
# Markdown

A simple Elixir [Markdown][1] to HTML conversion library.

Implemented entirely as a [NIF][2] binding to the [Hoedown][3] library.

## Usage

```iex
iex> Markdown.to_html "# Hello World"
"<h1>Hello World</h1>\n"
iex> Markdown.to_html "http://elixir-lang.org/", autolink: true
"<p><a href=\"http://elixir-lang.org/\">http://elixir-lang.org/</a></p>\n"
```

### Options

* `:tables` - Enables Markdown Extra style tables (default: `false`)
* `:fenced_code` - Enables fenced code blocks (default: `false`)
* `:autolink` - Automatically turn URLs into links (default: `false`)

## TODO

* Use dirty schedulers [when available][4]
* Add more options to the NIF that are [available in Hoedown][5]
* ANSI renderer

[1]: http://daringfireball.net/projects/markdown/
[2]: http://www.erlang.org/doc/tutorial/nif.html
[3]: https://github.com/hoedown/hoedown
[4]: https://github.com/zambal/markdown/commit/792a414de074d96ac47b5b52d212b7197ce04462
[5]: https://github.com/hoedown/hoedown/blob/8ab482af918f6b5c296dca80f0c67e924c06b5c2/bin/hoedown.c#L49


text:
#include <string.h>
#include <stdint.h>

#include "erl_nif.h"

#include "buffer.h"
#include "document.h"
#include "html.h"

#define OUTPUT_UNIT 128

typedef struct {
  ERL_NIF_TERM atom_true;
  ERL_NIF_TERM atom_tables;
  ERL_NIF_TERM atom_autolink;
  ERL_NIF_TERM atom_fenced_code;
} markdown_priv;

static ERL_NIF_TERM
to_html(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) {
  ErlNifBinary input;
  ErlNifBinary output;

  markdown_priv* priv;
  ERL_NIF_TERM options;
  ERL_NIF_TERM term;
  const ERL_NIF_TERM* tuple;
  int tuple_size;
  unsigned int extensions;

  hoedown_buffer* ob;
  hoedown_document* document;
  hoedown_renderer* renderer;

  if (enif_inspect_binary(env, argv[0], &input) == 0) {
    return enif_make_badarg(env);
  }

  if (input.size < 1) {
    return argv[0];
  }

  priv = enif_priv_data(env);
  options = argv[1];
  extensions = 0;

  while (enif_get_list_cell(env, options, &term, &options) != 0) {
    if (enif_get_tuple(env, term, &tuple_size, &tuple) != 0) {
      if (tuple_size > 1) {
        if (enif_compare(tuple[0], priv->atom_tables) == 0) {
          if (enif_compare(tuple[1], priv->atom_true) == 0) {
            extensions |= HOEDOWN_EXT_TABLES;
            continue;
          }
        }

        if (enif_compare(tuple[0], priv->atom_autolink) == 0) {
          if (enif_compare(tuple[1], priv->atom_true) == 0) {
            extensions |= HOEDOWN_EXT_AUTOLINK;
            continue;
          }
        }

        if (enif_compare(tuple[0], priv->atom_fenced_code) == 0) {
          if (enif_compare(tuple[1], priv->atom_true) == 0) {
            extensions |= HOEDOWN_EXT_FENCED_CODE;
            continue;
          }
        }
      }
    }
  }

  ob = hoedown_buffer_new(OUTPUT_UNIT);
  renderer = hoedown_html_renderer_new(0, 0);
  document = hoedown_document_new(renderer, extensions, 16);
  hoedown_document_render(document, ob, (uint8_t*) input.data, input.size);

  enif_release_binary(&input);
  hoedown_html_renderer_free(renderer);
  hoedown_document_free(document);

  enif_alloc_binary(ob->size, &output);
  memcpy(output.data, ob->data, ob->size);
  hoedown_buffer_free(ob);

  return enif_make_binary(env, &output);
}

static ErlNifFunc funcs[] = {
  { "to_html", 2, to_html }
};

static int
load(ErlNifEnv* env, void** priv, ERL_NIF_TERM info) {
  markdown_priv* data = enif_alloc(sizeof(markdown_priv));
  if (data == NULL) {
    return 1;
  }

  data->atom_true = enif_make_atom(env, "true");
  data->atom_tables = enif_make_atom(env, "tables");
  data->atom_autolink = enif_make_atom(env, "autolink");
  data->atom_fenced_code = enif_make_atom(env, "fenced_code");

  *priv = (void*) data;
  return 0;
}

static int
reload(ErlNifEnv* env, void** priv, ERL_NIF_TERM info) {
  return 0;
}

static int
upgrade(ErlNifEnv* env, void** priv, void** old_priv, ERL_NIF_TERM info) {
  return load(env, priv, info);
}

static void
unload(ErlNifEnv* env, void* priv) {
  enif_free(priv);
}

ERL_NIF_INIT(Elixir.Markdown, funcs, &load, &reload, &upgrade, &unload)


text:
defmodule Markdown do
  @moduledoc """
  Markdown to HTML conversion.
  """

  @on_load { :init, 0 }

  app = Mix.Project.config[:app]

  def init do
    path = :filename.join(:code.priv_dir(unquote(app)), 'markdown')
    :ok = :erlang.load_nif(path, 0)
  end

  @doc ~S"""
  Converts a Markdown document to HTML:

      iex> Markdown.to_html "# Hello World"
      "
Hello World
\n"
      iex> Markdown.to_html "http://elixir-lang.org/", autolink: true
      "
http://elixir-lang.org/
\n"

  Available output options:

  * `:tables` - Enables Markdown Extra style tables (default: `false`)
  * `:fenced_code` - Enables fenced code blocks (default: `false`)
  * `:autolink` - Automatically turn URLs into links (default: `false`)

  """
  @spec to_html(doc :: String.t) :: String.t
  @spec to_html(doc :: String.t, options :: Keyword.t) :: String.t
  def to_html(doc, options \\ [])

  def to_html(_, _) do
    exit(:nif_library_not_loaded)
  end
end


text:
defmodule MarkdownTest do
  use ExUnit.Case

  doctest Markdown

  test :tables do
    markdown = """
    |  J  |  O  |
    | --- | --- |
    |  S  |  É  |
    """

  	html = Markdown.to_html(markdown, tables: true)
    assert html =~ ~r/<table>/
  end

  test :autolink do
  	markdown = "http://devintorr.es/"
  	html = Markdown.to_html(markdown, autolink: true)
  	assert html =~ ~r[<a href="http://devintorr.es/">]
  end

  test :fenced_code do
  	markdown = """
  	```
  	Markdown.to_html(markdown)
  	```
  	"""

  	html = Markdown.to_html(markdown, fenced_code: true)
  	assert html =~ ~r/<code>/
  end
end


text: ExUnit.start


text:
## Public domain

The project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the [CC0 1.0 Universal public domain dedication][CC0].

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.

[CC0]: http://creativecommons.org/publicdomain/zero/1.0/

repo_name: unitedstates/uscode

repo_language: Python


text:
#!/usr/bin/env python

import sys
import os
import traceback
import pprint as pp

# name of the task comes first
task_name = sys.argv[1]

# parse any command line flags off
options = {}
args = sys.argv[2:]
for arg in args:
  if arg.startswith("--"):

    if "=" in arg:
      key, value = arg.split('=')
    else:
      key, value = arg, True

    key = key.split("--")[1]
    if value == 'True': value = True
    elif value == 'False': value = False
    options[key.lower()] = value

# store original raw args array after task name
options['argv'] = args

# depends on tasks/[task_name].py being present relative to this directory
sys.path.append("tasks")
import utils

try:
  __import__(task_name).run(options)
except Exception as exception:
  print utils.format_exception(exception)


text:
logbook
pytz
scrapelib
ipython
lxml


text:
## Parsing the US Code

A parser for the US Code's structure, and a work-in-progress parser for the US Code's full content.

### Setup

Create a virtual environment:

```bash
virtualenv virt
source virt/bin/activate
pip install -r requirements.txt
```

### Getting the structure of the Code

To output the hierarchy of the US Code to STDOUT, in JSON:

```bash
./run structure
```

The script will first download the

Options:

* `--year`: "uscprelim" (the default), or a specific year version of the Code (e.g. "2011")
* `--title`: Do only a specific title (e.g. "5", "5a", "25")
* `--sections`: Return a flat hierarchy of only titles and sections (no intervening layers)
* `--debug`: Output debug messages only, and no JSON output (dry run)
* `--force`: Force a re-download of the US Code. Use this flag if you're automatically running the script at an interval.

Example:

```json
[
  {
    "level": "title",
    "name": "GENERAL PROVISIONS",
    "number": "1",
    "subparts": [
      {
        "level": "chapter",
        "name": "RULES OF CONSTRUCTION",
        "number": "1",
        "subparts": [
          {
            "citation": "usc/1/1",
            "level": "section",
            "name": "Words denoting number, gender, and so forth",
            "number": "1"
          },
          {
            "citation": "usc/1/2",
            "level": "section",
            "name": "\u201cCounty\u201d as including \u201cparish\u201d, and so forth",
            "number": "2"
          },
          ...
        ]
      }
    ]
  }
]
```

### Getting the content of the Code (work-in-progress)

To get at the content of the Code:

* Run `download/gpolocator.sh 2011` to download all GPO Locator files for 2011.
* Run download/pdf.sh to download all pdf files for 2011.

Run the debug script with the title as the first argument and the offset of the parsed node in the parsed title (yes, that makes no sense--just enter a number, like 3).

```bash
source virt/bin/activate # if not already activated
./run debug title=[title] offset=[offset]
```

So to view title 11, section 1, which is the definitions section of the bankruptcy code, run:

```bash
./run debug title=11 offset=3
```

## Public domain

This project is [dedicated to the public domain](LICENSE). As spelled out in [CONTRIBUTING](CONTRIBUTING.md):

> The project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the [CC0 1.0 Universal public domain dedication](http://creativecommons.org/publicdomain/zero/1.0/).

> All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.


text:
# Download and unzip all uscode files for a year.
# Run it from the uscode folder. Specify a year on
# the command line eg: download/gpolocator.sh 2011
DIR=`pwd`"/data"
DEST="$DIR/uscode.house.gov/zip/$1/"
#echo $DIR
#echo $DEST
mkdir -P $DIR
wget -m -l1 -P $DIR http://uscode.house.gov/zip/$1
cd $DEST
for filename in `ls`; do unzip $filename; done
find . -type f -name "*zip" -delete
cd $DIR


text:
# Download and unzip all uscode pdf files for 2011.
DIR=~/data/
mkdir -P $DIR
wget -m -l1 -P $DIR http://uscode.house.gov/pdf/2011/


text:
# import unittest
# import pprint

# class TestRelatedCitation(unittest.TestCase):

#     maxDiff = None

#     def test_parse_all(self):
#         for string, data in samples:
#             _data = parse(Lexer, Parser, ParserState, string)
#             import pdb;pdb.set_trace()
#             # pprint.pprint(data)
#             # pprint.pprint(_data)
#             #self.assertEqual(data, _data)

def main():
    from gpolocator import schemes

    import pdb;pdb.set_trace()
    schemes.Token('ll').get_schemes()

    for k, v in schemes._schemes_lists.items():
        print v
        for t in v:
            en = schemes.Enum(t)
            print en
            try:
                print en.get_schemes()
                print en.get_ordinality()
            except Exception as e:
                print 'got', e
                import pdb;pdb.set_trace()
                en.get_ordinality()
            # print en.get_schemes()

if __name__ == '__main__':
    main()


text:
import os, json
import utils
import uscode

def run(options):
  title_number = options.get('title', None)
  # section = options.get('section', None)
  year = options.get('year', 2011) # default to 2011 for now

  if not title_number:
    utils.log("Supply a 'title' argument to parse a title.")
    return

  filename = utils.title_filename(title_number, year)
  if not os.path.exists(filename):
    utils.log("This title has not been downloaded.")

  title = uscode.title_for(filename)

  sections = title.sections()

  count = 0
  for section in sections:
    section_number = section.enum()
    print "[%s USC %s] Parsing..." % (title_number, section_number)

    # this could probably be black boxed a little further?
    bb = section.body_lines()
    xx = uscode.GPOLocatorParser(bb)
    qq = xx.parse()
    output = qq.json()

    utils.write(
      json.dumps(output, sort_keys=True, indent=2),
      uscode_output(year, title_number, section_number)
    )

    count += 1

  print "\nParsed %s sections of title %s." % (count, title_number)

def uscode_output(year, title, section):
  return "%s/%s/%s/%s.json" % (utils.output_dir(), year, title, section)


text:
import os
import sys
import shutil
from os.path import join
import subprocess

from logbook import Logger
from uscode import File
import utils

logger = Logger('debug')

def run(options):
    argv = options["argv"]

    args = [
        ('2011', '2006 Edition and Supplement V (2011)'),
        ('2010', '2006 Edition and Supplement IV (2010)'),
        ('2009', '2006 Edition and Supplement III (2009)'),
        ('2008', '2006 Edition and Supplement II (2008)'),
        ('2007', '2006 Edition and Supplement I (2007)'),
        ('2006', '2006 Edition (2006)'),
        ('2005', '2000 Edition and Supplement V (2005)'),
        ('2004', '2000 Edition and Supplement IV (2004)'),
        ('2003', '2000 Edition and Supplement III (2003)'),
        ('2002', '2000 Edition and Supplement II (2002)'),
        ('2001', '2000 Edition and Supplement I (2001)'),
        ('2000', '2000 Edition (2000)'),
        ('1999', '1994 Edition and Supplement V (1999)'),
        ('1998', '1994 Edition and Supplement IV (1998)'),
        ('1997', '1994 Edition and Supplement III (1997)'),
        ('1996', '1994 Edition and Supplement II (1996)'),
        ('1995', '1994 Edition and Supplement I (1995)'),
        ('1994', '1994 Edition (1994)'),
        ]

    path = '/home/thom/code/uscode-git'
    try:
        logger.info('Removing tree: %r' % path)
        shutil.rmtree(path)
    except OSError:
        pass

    # Create the repo again.
    logger.info('Creating the git repo.')
    subprocess.check_call('cd /home/thom/code && mkdir -p ' + path, shell=True)
    subprocess.check_call('touch %s/README' % path, shell=True)
    subprocess.check_call('cd %s && git init && git add .' % path, shell=True)
    subprocess.check_call('cd %s && git remote add origin git@github.com:unitedstates/uscode-git.git' % path, shell=True)

    succeeded = 0
    failed = 0
    failed_objs = []
    for arg in reversed(args):

        for title in range(1, 51):

            year, commit_msg = arg
            logger.info('Writing files for %s ...' % year)
            filename = utils.title_filename(title, year)

            title_path = join(path, str(title))

            try:
                os.mkdir(title_path)
            except OSError:
                pass
            else:
                logger.info('created dir %s' % title_path)

            try:
                fp = open(filename)
            except IOError as e:
                logger.warning('No such file: %r: %r' % (filename, e))

            try:
                gpo_file = File(fp)
            except Exception as e:
                logger.critical('The parser failed on %r' % filename)

            for section in gpo_file.sections():
                try:
                    msg = 'Trying to parse %r' % section
                    # logger.info(msg)
                except Exception as e:
                    logger.warning('Something terrible happend.')
                    failed_objs.append((section, e))
                    continue
                try:
                    tree = section.as_tree()
                except Exception as e:
                    logger.critical('Parse failed! %r' % e)
                    failed += 1
                    failed_objs.append((section, e))
                else:
                    succeeded += 1

                section_path = join(title_path, '%s.txt' % str(section.enum()))
                with open(section_path, 'w') as f:
                    tree.filedump(f)

        # import pdb;pdb.set_trace()

        # try:
        #     shutil.rmtree(path)
        # except OSError:
        #     pass
        # os.mkdir(join(path, 'title%d' % title))
        #js = qq.json()
        #fson.dump(js, join(path, 'title%d' % title))
            commit_msg = 'Title %d: %s' % (title, commit_msg)
            cmd = 'cd %r && git add . && git commit -am"%s"' % (path, commit_msg)
            logger.info('Running %r' % repr(cmd))
            try:
                out = subprocess.check_output(cmd, shell=True)
            except subprocess.CalledProcessError as e:
                print e
    # subprocess.check_call('cd /home/thom/code && octogit create '
    #                       '11USC101 "11 USC 101 in git"', shell=True)
    # subprocess.check_call('cd %s && git push origin master' % path, shell=True)

if __name__ == '__main__':
    import sys
    main(sys.argv[1:])


text:
# Downloads and uses the XHTML version of the US Code to extract a table of contents.
# Defaults to USCprelim.
#
# Outputs JSON to STDOUT. Run and save with:
#  ./run structure > structure.json
#
# options:
#   year: "uscprelim" (the default), or a specific year version of the Code (e.g. "2011")
#   title: Do only a specific title (e.g. "5", "5a", "25")
#   sections: Return a flat hierarchy of only titles and sections (no intervening layers)
#   debug: Output debug messages only, and no JSON output (dry run)
#   force: Force a re-download of the US Code for the given year (script defaults to caching if the directory for a year is present)

import glob, re, lxml.html, json, sys, os

import utils

import HTMLParser
pars = HTMLParser.HTMLParser()

section_symbol = u'\xa7'

def run(options):
  year = options.get("year", "uscprelim") # default to USCprelim

  # optional: don't print json out, just --debug information
  debug = options.get('debug', False)

  # optional: limit to a specific --title
  title = options.get("title", None)
  if title:
    # appendix cites may look like "5a" but we need "05a" to match the file
    if title.endswith("a"):
      title = "*usc%02da" % int(title[0:-1])
    else:
      title = "*usc%02d" % int(title)
  else:
    title = "*usc*"

  # sync XHTML to disk as needed (cache by default)
  download_usc(year, options)

  filenames = glob.glob("data/uscode.house.gov/xhtml/" + year + "/%s.htm" % title)
  filenames.sort()

  # optional: --limit to a number of titles
  limit = options.get("limit", None)
  if limit:
    filenames = filenames[0:int(limit)]

  # optional: only retrieve titles and --sections, nothing in between
  sections_only = options.get("sections", False)

  # Output and intermediary data structures.

  TOC = [ ]
  path = None

  # Loop through all titles of the code...
  for fn in filenames:
    match = re.search(r"usc(\d+a?)\.htm$", fn, re.I)
    if not match: continue

    # extract title, to have on hand when parsing sections, and debug output
    title = match.groups(1)[0]
    if debug:
      print "[%s] Processing title..." % title

    # Parse the XHTML file...
    dom = lxml.html.parse(fn)

    # The file structure is flat. Loop through the XML
    # nodes in this title.
    for n in dom.find("body/div"):
      # Look for comments of the form <!-- expcite:... -->
      # This tells us the current table of contents location for the following <h3>.
      m = re.match(ur"<!-- expcite:(.*\S)\s*-->", unicode(n))
      if m:
        # This is a "!@!"-separated string giving the table-of-contents
        # path to each section as we see it.
        expcite = m.group(1)

        # These comments have HTML entities. Replace them with unicode.
        expcite = expcite.replace(" ", " ")
        expcite = pars.unescape(expcite)

        # Parse the table of contents path.
        path = parse_expcite(expcite)

      elif n.tag == "h3":
        # All headings are h3s. Check if it starts with the curly S section symbol.
        h3 = n.text_content()
        if not h3.startswith(section_symbol): continue

        # The most recent expcite path is the TOC location of this section.
        if not path: raise Exception("h3 without path")

        # Insert the section into our TOC structure.
        parse_h3(path, h3, TOC, title, sections_only)

        # Clear so we don't reuse the path on the next h3.
        path = None

  # Sort the titles (take into account appendix notation).
  TOC.sort(key = lambda title : (int(title[0][1].replace("a", "")), title[0][1]))

  # Reformat the output.
  TOC = [reformat_structure(title) for title in TOC]

  # Write output in JSON to stdout.
  if debug:
    print "\n(dry run only, not outputting)"
  else:
    json.dump(TOC, sys.stdout, indent=2, sort_keys=True, check_circular=False)

def parse_expcite(expcite):
  path = expcite.split("!@!")

  # Parse each part of the path:
  for i in xrange(len(path)):
    if re.match(r"\[.*-(REPEALED|RESERVED|OMITTED|TRANSFERRED)\]\s*$", path[i], re.I):
      # This part is repealed. No need to process this path at all.
      path = None
      break

    m = re.match(r"(TITLE|SUBTITLE|CHAPTER|SUBCHAPTER|PART|SUBPART|DIVISION) ([^\-]+)-(.*)$|Secs?\. (.*)", path[i], re.I)
    if not m:
      # Some random text string. It's a part of a title with no level specifier.
      # We'll call it a "heading" level. In Title 50a, there are just names of
      # acts under the title, no apparent chapter.
      path[i] = ("heading", None, path[i])

    elif m.group(1):
      # Matches TITLE|CHAPTER...
      # Store as (TITLE|CHAPTER, NUMBER, NAME)
      # Replace en-dashes in the number with simple dashes, as we do with section numbers.

      path[i] = (m.group(1).lower(), m.group(2).replace(u"\u2013", "-"), m.group(3))

      # Reformat title appendices: XXX, APPENDIX => XXXa.
      if i == 0 and path[i][0] == "title" and ", APPENDIX" in path[i][1]:
        path[i] = (path[i][0], path[i][1].replace(", APPENDIX", "a"), path[i][2] + " (APPENDIX)")
      elif i == 0 and path[i][0] == "title" and path[i][2] == "APPENDIX": # titles 5, 18 look like this
        path[i] = (path[i][0], path[i][1] + "a", path[i][2])

    elif m.group(4) and i == len(path) - 1:
      # Matches a section number or range of sections.
      # We'll get this information from the next <h3> element, which also has the
      # section title.
      path.pop()

    else:
      raise Exception("Invalid expcite?")

  return path

def parse_h3(path, h3, TOC, title, sections_only = False):
  # Skip sections that are just placeholders.
  if re.match(section_symbol + section_symbol + r"?(.*?)\. (Repealed.*|Transferred|Omitted)(\.|$)", h3):
    # This is for multiple sections, which are always repealed, or
    # repealed/transferred sections.
    return

  # Reformat section numbers. Replace en-dashes with simple dashes, as we do with chapter etc. numbers.
  h3 = h3.replace(u"\u2013", "-")

  # Parse the section number and description, and add that to the path.
  m = re.match(section_symbol + r"(.*?)\.? (.*)", h3)
  if not m: raise Exception("Could not parse: " + h3)

  number = m.group(1)
  name = m.group(2)
  citation = citation_for(title, number)
  path.append( ("section", number, name, citation))

  # Add the new path into the TOC, making a structure like:
  #  [ ( (title, 17, Copyright), [
  #       ... sub parts ..
  #      ])
  #  ]
  _toc = TOC
  for p in path:

    # allow the caller to discard levels that are not normally cited
    if sections_only and (p[0] not in ['title', 'section']):
      continue

    if len(_toc) == 0 or _toc[-1][0] != p:
      _toc.append( (p, []) )
    _toc = _toc[-1][1] # move in

def reformat_structure(entry):
  ret = {
    "level": entry[0][0],
    "number": entry[0][1],
    "name": entry[0][2],
  }

  # sections have an additional citation field
  if entry[0][0] == "section":
    ret["citation"] = entry[0][3]

  if len(entry[1]):
    ret["subparts"] = [reformat_structure(e) for e in entry[1]]
  return ret

def citation_for(title, number):
  # title may be 0-prefixed, ditch for purposes of citation
  if title.startswith("0"):
    t = title[1:]
  else:
    t = title
  return "usc/%s/%s" % (t, number)

def download_usc(year, options):
  debug = options.get("debug", False)

  dest_dir = "data/uscode.house.gov/xhtml/%s" % year

  if os.path.isdir(dest_dir) and not options.get("force", False):
    if debug: print "Cached, not downloading again"
    return # assume it's downloaded

  if debug: print "Downloading US Code XHTML for %s" % year
  utils.mkdir_p(dest_dir)
  os.system("rm %s/*" % dest_dir)
  os.system("wget -q -m -l1 -P %s http://uscode.house.gov/xhtml/%s" % ("data", year))


text:
#! /home/thom/ve/2.7/bin/python
import sys

from logbook import Logger

from uscode import File
from utils import title_filename

logger = Logger('debug')

def run(options):
    filename = title_filename(int(options["title"]), '2011')
    fp = open(filename)

    gpo_file = File(fp)
    succeeded = 0
    failed = 0
    for section in gpo_file.sections():
        logger.info('Trying to parse %r' % section)
        try:
            tree = section.as_tree()
        except Exception as e:
            logger.warning('  .. parse failed: %r' % e)
            failed += 1
        else:
            succeeded += 1

    logger.info('Number that succeeded: %d' % succeeded)
    logger.info('Number that failed: %d' % failed)
    logger.info('Percent success: %f' % (1 - (1.0 * failed / succeeded)))

    # ss = ff[int(options["offset"])].instance
    # bb = ss.body_lines()
    # xx = GPOLocatorParser(bb)
    # qq = xx.parse()
    # qq.tree()
    # js = qq.json()

    import pdb;pdb.set_trace()

    # xx = [x.instance for x in gg]

    # import pdb;pdb.set_trace()
    # for doc in gg:
    #     x = doc.instance
    #     print x
    #     import pdb;pdb.set_trace()

if __name__ == '__main__':
    main()


text:
# Downloads and uses the XML version of the US Code to extract a table of contents.
#
# Outputs JSON to STDOUT. Run and save with:
#  ./run structure_xml > structure.json
#
# options:
#   title: Do only a specific title (e.g. "5", "5a", "25")
#   sections: Return a flat hierarchy of only titles and sections (no intervening layers)
#   debug: Output debug messages only, and no JSON output (dry run)
#   force: Force a re-download of the US Code

import glob, re, lxml.etree, lxml.html, json, sys, os, os.path, urllib, zipfile

import utils

import HTMLParser
pars = HTMLParser.HTMLParser()

section_symbol = u'\xa7'

ns = {
  "uslm": "http://xml.house.gov/schemas/uslm/1.0"
}

def run(options):
  # optional: don't print json out, just --debug information
  debug = options.get('debug', False)

  # optional: limit to a specific --title
  title = options.get("title", None)
  if not title:
    title = "*"
  else:
    title = "xml_usc" + title + "@*"

  # sync XML to disk as needed (cache by default)
  download_usc(options)

  filenames = glob.glob("data/uscode.house.gov/xml/%s.zip" % title)
  filenames.sort()

  # optional: --limit to a number of titles
  limit = options.get("limit", None)
  if limit:
    filenames = filenames[0:int(limit)]

  # optional: only retrieve titles and --sections, nothing in between
  sections_only = options.get("sections", False)

  # process titles

  TOC = [ ]

  for fn in filenames:
    zf = zipfile.ZipFile(fn, "r")
    xmlbody = zf.read(os.path.basename(fn).replace("xml_", "").replace(".zip", ".xml"))
    #print xmlbody
    dom = lxml.etree.fromstring(xmlbody)
    titlenode = dom.xpath("uslm:main/uslm:title", namespaces=ns)[0]
    proc_node(titlenode, TOC, [], sections_only)

  # Sort the titles (take into account appendix notation).
  TOC.sort(key = lambda title : (int(title["number"].replace("a", "")), title["number"]))

  # Write output in JSON to stdout.
  if debug:
    print "\n(dry run only, not outputting)"
  else:
    json.dump(TOC, sys.stdout, indent=2, sort_keys=True, check_circular=False)

def proc_node(node, parent, path, sections_only):
  # Form the node for this title/chapter/.../section.

  remove_footnotes(node.xpath("uslm:heading", namespaces=ns)[0])
  entry = {
    "level": lxml.etree.QName(node.tag).localname,
    "number": unicode(node.xpath("string(uslm:num/@value)", namespaces=ns)),
    "name": unicode(node.xpath("string(uslm:heading)", namespaces=ns)),
  }
  if entry["level"] == "level": entry["level"] = "heading"

  # To compare with our older HTML scraper, put these lines back in to normalize the content a bit.
  #entry["name"] = entry["name"].replace(u"\u2019", "'") # curly right apostrophe => straight apostrophe (it's inconsistent, so for diffs: sed -i "s/\\\\u2019/'/g" structure_html.json)
  #entry["name"] = entry["name"].replace(u"\u202f", u"\u00a0") # narrow no-break space => no-break space (probably convert this to a space later on)
  #entry["name"] = entry["name"].replace(u"\u2013", "-") # replace en-dashes with simple hyphens
  #if u"\u00a7\u202f" in entry["number"]: return # the HTML converter misses these

  # Misformatting
  entry["number"] = entry["number"].replace(u"\u00a7\u202f", "") # section symbol plus narrow no-break space

  # Text reformatting.
  entry["name"] = entry["name"].strip() # TODO: Flag upstream, remove this line when fixed.
  entry["number"] = entry["number"].strip() # TODO: Flag upstream, remove this line when fixed.
  entry["name"] = entry["name"].replace(u"\u00ad", "") # remove soft hyphens
  entry["number"] = entry["number"].replace(u"\u2013", "-") # replace en-dashes with simple hyphens
  if entry["number"] == "": entry["number"] = None

  # Don't record structure of phantom parts.
  if entry["name"] in ("]", "Repealed", "Reserved", "Reserved]", "Omitted", "Omitted]", "Transferred", "Transferred]", "Omitted or Transferred", "Vacant]"):
    return
  if re.match(r"(Repealed.*|Transferred|Omitted|Renumbered .*\])(\.|$)", entry["name"]):
    return

  # Make an array of level numbering in the path to this section.
  # (To compare with the old HTML output, disable everything but section-level citations.)

  if entry['level'] != None and entry['number'] != None:
    path = path + [(entry['level'], entry['number'])]
  else:
    path = path + [None] # flag that this level and descendants do not have a path

  if entry["level"] == "section":
    entry["citation"] = "usc/%s/%s" % (path[0][1], entry["number"]) # title number & section number only
  elif entry["level"] == "chapter":
    # chapter numbering is unique within a title, like sections, but may be split across
    # divisions and other levels beneath the title level. since finding chapter citations
    # is important, pop their scope up so their citation values are predictable without
    # having to know its intermediate levels of embedding.
    entry["citation"] = "usc/title/%s/chapter/%s" % (path[0][1], entry["number"])
  elif None in path:
    # can't create a citation if there is an unnumbered level on the path
    pass
  else:
    # for other levels, encode them beneath the title as a path through the numbers
    entry["citation"] = "usc/%s" % "/".join("%s/%s" % p for p in path)

  # Debugging helper.
  #if entry.get("citation") == "usc/4/107":
  #  print lxml.etree.tostring(node)
  #  print entry

  # Recurse into children.

  children = []

  if entry["level"] != "section":
  	  # 25 USC 450l has a section within a section, so skip this processing because we never want bottom-half levels of structure
	  for child in node.xpath("uslm:title|uslm:subtitle|uslm:chapter|uslm:subchapter|uslm:part|uslm:subpart|uslm:division|uslm:level|uslm:section", namespaces=ns):
		proc_node(child, children, path, sections_only)

  if len(children):
    entry["subparts"] = children

  # Our older HTML scraper didn't include levels without subparts, except sections.
  #if not len(children) and entry["level"] != "section": return

  # Pop back up.

  if sections_only and entry["level"] not in ['title', 'section']:
    # We're only interested in these two levels. Flatten the hierarchy.
    parent.extend(children)
  else:
    # Add this entry to the parent's list of children.
    parent.append(entry)

def remove_footnotes(node):
  # Remove footnote text and footnote markers from the heading text.
  # lxml makes removing nodes tricky because the tail text gets removed too, but it needs to be moved.
  def filter_nbsp(t):
    if not t: return ""
    if t[-1] in (u"\u00a0", u"\u202f"): t = t[:-1] # footnotes are often preceded by a non-breaking space which we can remove
    return t
  for n in node.xpath("uslm:note|uslm:ref[@class='footnoteRef']", namespaces=ns):
    if n.tail:
      if n.getprevious() != None:
        n.getprevious().tail = filter_nbsp(n.getprevious().tail) + n.tail
      else:
        n.getparent().text = filter_nbsp(n.getparent().text) + n.tail
    n.getparent().remove(n)

def download_usc(options):
  debug = options.get("debug", False)

  dest_dir = "data/uscode.house.gov/xml"
  utils.mkdir_p(dest_dir)

  base_url = "http://uscode.house.gov/download/"
  index_page = lxml.html.parse(urllib.urlopen(base_url + "download.shtml")).getroot()
  for anode in index_page.xpath('//a[.="[XML]"]'):
    if "uscAll@" in anode.get("href"): continue # skip the all-titles archive
    if "xml_usc34@" in anode.get("href"): continue # title 34 doesn't exist (was repealed)
    source_url = base_url + anode.get("href")
    dest_path = dest_dir + "/" + os.path.basename(anode.get("href"))
    if os.path.exists(dest_path) and not options.get("force", False):
      continue
    os.system("wget -O %s %s" % (dest_path, source_url))


text:
import os
import sys
import shutil
from os.path import join
import subprocess

from uscode.parser import getlines
from uscode.grouper import group
from uscode.structure import GPOLocatorParser
import utils
import fson

def run(options):
    argv = options["argv"]

    args = [
        ('2011', '2006 Edition and Supplement V (2011)'),
        ('2010', '2006 Edition and Supplement IV (2010)'),
        ('2009', '2006 Edition and Supplement III (2009)'),
        ('2008', '2006 Edition and Supplement II (2008)'),
        ('2007', '2006 Edition and Supplement I (2007)'),
        ('2006', '2006 Edition (2006)'),
        ('2005', '2000 Edition and Supplement V (2005)'),
        ('2004', '2000 Edition and Supplement IV (2004)'),
        ('2003', '2000 Edition and Supplement III (2003)'),
        ('2002', '2000 Edition and Supplement II (2002)'),
        ('2001', '2000 Edition and Supplement I (2001)'),
        ('2000', '2000 Edition (2000)'),
        ('1999', '1994 Edition and Supplement V (1999)'),
        ('1998', '1994 Edition and Supplement IV (1998)'),
        ('1997', '1994 Edition and Supplement III (1997)'),
        ('1996', '1994 Edition and Supplement II (1996)'),
        ('1995', '1994 Edition and Supplement I (1995)'),
        ('1994', '1994 Edition (1994)'),
        ]

    path = '/home/thom/code/11USC101'
    try:
        shutil.rmtree(path)
    except OSError:
        pass
    subprocess.check_call('cd /home/thom/code && mkdir 11USC101', shell=True)
    subprocess.check_call('touch %s/README' % path, shell=True)
    subprocess.check_call('cd %s && git init && git add .' % path, shell=True)

    for arg in reversed(args):
        year, msg = arg
        print 'Committing', year, '...'
        title = int(argv[0])
        offset = int(argv[1])
        filename = utils.title_filename(title, year)
        fp = open(filename)
        lines = getlines(fp)
        gg = group(lines)

        ss = gg[offset].instance
        bb = ss.body_lines()
        xx = GPOLocatorParser(bb)
        qq = xx.parse()

        # try:
        #     shutil.rmtree(path)
        # except OSError:
        #     pass
        # os.mkdir(join(path, 'title%d' % title))
        js = qq.json()
        fson.dump(js, join(path, 'title%d' % title))
        cmd = 'cd %r && git add . && git commit -am"%s"' % (path, msg)
        print 'Running', repr(cmd)
        try:
            out = subprocess.check_output(cmd, shell=True)
        except subprocess.CalledProcessError as e:
            print e
    # subprocess.check_call('cd /home/thom/code && octogit create '
    #                       '11USC101 "11 USC 101 in git"', shell=True)
    # subprocess.check_call('cd %s && git push origin master' % path, shell=True)

if __name__ == '__main__':
    import sys
    main(sys.argv[1:])


text:
import os, errno, sys, traceback
import re, htmlentitydefs
import pprint

from pytz import timezone
import datetime, time

# scraper should be instantiated at class-load time, so that it can rate limit appropriately

import scrapelib
scraper = scrapelib.Scraper(requests_per_minute=120, follow_robots=False, retry_attempts=3)

# manage input and output dirs

def output_dir():
  return "data/output"

def input_dir():
  return "data/uscode.house.gov/zip"

def title_filename(title, year=2011):
  year = str(year)
  return os.path.join(input_dir(), year, 'usc%02d.%02d' % (int(title), int(year[2:])))

# general purpose

def log(object):
  if isinstance(object, str):
    print object
  else:
    pprint.pprint(object)

def format_datetime(obj):
  if isinstance(obj, datetime.datetime):
    return obj.replace(microsecond=0, tzinfo=timezone("US/Eastern")).isoformat()
  elif isinstance(obj, str):
    return obj
  else:
    return None

def download(url, destination, force=False):
  cache = os.path.join(cache_dir(), destination)

  if not force and os.path.exists(cache):
    log("Cached: (%s, %s)" % (cache, url))
    with open(cache, 'r') as f:
      body = f.read()
  else:
    try:
      log("Downloading: %s" % url)
      response = scraper.urlopen(url)
      body = str(response)
    except scrapelib.HTTPError as e:
      log("Error downloading %s:\n\n%s" % (url, format_exception(e)))
      return None

    # don't allow 0-byte files
    if (not body) or (not body.strip()):
      return None

    # cache content to disk
    write(body, cache)

  return unescape(body)

def write(content, destination):
  mkdir_p(os.path.dirname(destination))
  f = open(destination, 'w')
  f.write(content)
  f.close()

# mdir -p in python, from:
# http://stackoverflow.com/questions/600268/mkdir-p-functionality-in-python
def mkdir_p(path):
  try:
    os.makedirs(path)
  except OSError as exc: # Python >2.5
    if exc.errno == errno.EEXIST:
      pass
    else:
      raise

# taken from http://effbot.org/zone/re-sub.htm#unescape-html
def unescape(text):

  def remove_unicode_control(str):
    remove_re = re.compile(u'[\x00-\x08\x0B-\x0C\x0E-\x1F\x7F]')
    return remove_re.sub('', str)

  def fixup(m):
    text = m.group(0)
    if text[:2] == "&#":
      # character reference
      try:
        if text[:3] == "&#x":
          return unichr(int(text[3:-1], 16))
        else:
          return unichr(int(text[2:-1]))
      except ValueError:
        pass
    else:
      # named entity
      try:
        text = unichr(htmlentitydefs.name2codepoint[text[1:-1]])
      except KeyError:
        pass
    return text # leave as is

  text = re.sub("&#?\w+;", fixup, text)
  text = remove_unicode_control(text)
  return text

# doesn't actually use the passed-in exception, but, er, I feel like I should pass it in anyway
def format_exception(exception):
  exc_type, exc_value, exc_traceback = sys.exc_info()
  return "\n".join(traceback.format_exception(exc_type, exc_value, exc_traceback))


text:
# -*- coding: utf-8 -*-
"""
Created on Wed Feb 19 19:24:15 2014
The purpose of this script is to parse the mapping of Public Law sections to U.S. Code sections.
This file parses HTML files such as http://uscode.house.gov/table3/111_148.htm
The output is a list of references, currently printing to standard output.
@author: Pablo, @williampli
"""

from __future__ import division
from bs4 import BeautifulSoup
import re
import sys

def parseTable(f):

    soup = BeautifulSoup(f)
    f.close()

    t = soup.find('table')
    t3 = t.find_all('tr')

    sections = []
    for row in t3:
        cols = map(lambda x: x.get_text(),row.find_all("td"))
        if len(cols) > 4:
            sections.append((cols[2],cols[3]))
    return sections

if __name__ == "__main__":
    args = sys.argv
    if len(args) != 2:
        print 'Usage: python ParseTable.py inputHTMLfile.htm'
        sys.exit()
    filename = sys.argv[1]
    f = file(filename,'r')
    sections = parseTable(f)
    print sections


text:
This subdirectory contains the tools for extracting information from "Table III". As noted on http://uscode.house.gov/table3/table3explanation.htm, Table III contains information about the classification of public laws to the United States Code. This information is also available in PDF form on the U.S. Code website, but the HTML pages seemed easier to parse.

The key files are:

* table3_scraper.py
* ParseTable.py

table3_scraper.py does the following:

1. Parse http://uscode.house.gov/table3/table3years.htm to extract links to year-specific pages with laws, e.g. http://uscode.house.gov/table3/year1950.htm
2. Parse each of the year-specific pages for extract links to a page for each law, e.g. http://uscode.house.gov/table3/1950_654.htm
3. Download the full HTML of the year-specific page.

The years downloaded are specified in a set called `years` in the script. To limit the number of files downloaded, you can set the `LIMIT_SUBSUBRELEASES` to `True` and `LIMIT` to the desired total number of files.

---

ParseTable.py does the following:

1. Take, as input, a downloaded law-specific HTML file from the final step of table3_scraper. The command is `python ParseTable.py [inputHTMLfile]
2. Extract the mapping of public law sections to U.S. Code sections.


text:
#!/usr/bin/python
import os, sys
import httplib2
from lxml import etree
from StringIO import StringIO
import re
import simplejson

# Table Three Scraper http://uscode.house.gov/table3/table3years.htm
# The scrapers grab the URLs for each year from 1789 to 2011, go one directory down to grab the directory, then go one directory below and grab the whole page.  THIS CODE TAKES A WHILE TO RUN.  It may be better to tweak just for the years you want.  Also, could use some refactoring, e.g. merge some of the functions.

# This script downloads files into the current directory

# GLOBAL VARIABLES

# Specify the years you want in a set
years = { 1950 }

# for testing purposes, the number of files downloaded can be limited.
LIMIT_SUBSUBRELEASES = False
LIMIT = 5

def mainscraper(content): #function to parse Table 3 website
	doc = etree.parse(StringIO(content), parser=etree.HTMLParser())
	releases = []
	subreleases = []
	for element in doc.xpath('//div[@class="alltable3years"]'):   #Could also use "alltable3statutesatlargevolumes"
		for d_element in element.xpath('span'):
			text = d_element.xpath('a')[0].text
			unitext = unicode(text).encode(sys.stdout.encoding, 'replace')
			for m_element in d_element.xpath('a'):
				addy = m_element.attrib['href']
				year = addy.replace( 'year', '' )
				year = year.replace( '.htm', '' )
				if int( year ) in set( years ):
					url = "http://uscode.house.gov/table3/" + addy
				        #print unitext, url
				        #releases += [(unitext, url)]
					subreleases += add_subrelease(url)
					#return subreleases
				else:
					pass
	return subreleases #releases, subreleases

def subscraper(content): #function to parse Table 3 website
	doc = etree.parse(StringIO(content), parser=etree.HTMLParser())
	subsubreleases = []
	releases = []
	for element in doc.xpath('//div[@class="yearmaster"]'):   #Could also use "statutesatlargevolumemasterhead"
		for d_element in element.xpath('span'):
			text = d_element.xpath('a')[0].text
			unitext = unicode(text).encode(sys.stdout.encoding, 'replace')
			for m_element in d_element.xpath('a'):
				addy = m_element.attrib['href']
				url = "http://uscode.house.gov/table3/" + addy
				print addy
				#print text, url
				#releases += [(text, url)]
				subsubreleases.append( add_subsubrelease(url) )
				if LIMIT_SUBSUBRELEASES and len( subsubreleases ) == LIMIT:
					return subsubreleases

	return	subsubreleases

def add_release(url): #function grab main page data
	http = httplib2.Http('/tmp/httpcache')
	response, content = http.request(url)
	if response.status != 200:
	    sys.stderr.write('Error, returned status: %s\n' % response.status)
	    sys.exit(1) #bomb out, non-zero return indicates error
	#print content
	return mainscraper(content)

def add_subrelease(url): #function to grab sub page data
	http = httplib2.Http('/tmp/httpcache')
	response, content = http.request(url)
	if response.status != 200:
	    sys.stderr.write('Error, returned status: %s\n' % response.status)
	    sys.exit(1) #bomb out, non-zero return indicates error
	#print content
	return subscraper(content)

def add_subsubrelease(url): #function to grab sub, sub page data
	http = httplib2.Http('/tmp/httpcache')
	response, content = http.request(url)
	if response.status != 200:
	    sys.stderr.write('Error, returned status: %s\n' % response.status)
	    sys.exit(1) #bomb out, non-zero return indicates error
	# print content
	return url, content

def main():
	dataset = []
	x = add_release("http://uscode.house.gov/table3/table3years.htm") #Could also use "/alltable3statutesatlargevolumes.html"
	for filename, html_string in x:
		final_pagename = filename.split('/')[-1]
		with open( final_pagename, 'w' ) as f:
			f.write( html_string )
		sys.stderr.write( "Wrote %s\n" % ( final_pagename ) )

if __name__ == '__main__':
	main()