Pingar Helps LegiNation Automatically Track and Analyze Hundreds of Thousands of Constantly Changing Unstructured Government Documents

Business Challenge

Large volumes of continuously updated unstructured data in a wide variety of formats needed to be automatically transformed into structured data in order to be useful for analyzing trends and tracking topic-specific information.

Business Solution

Pingar provides automatic extraction and application of content-specific metadata. The process is automatic, scalable and could be quickly implemented to meet project timelines.

Unstructured Government Data gets Organized to Mobilize Voters

Across the 50 United States, Puerto Rico and Washington DC, several hundred thousand proposed laws, or legislative bills, are introduced during each legislative session. As each bill moves through the process of becoming a law, it undergoes an extensive series of reviews and at each review stage, multiple edits are made.

The proposed bills, which can be extremely large documents, must be published and each change to the bill must be made available to the public, legislators and advocacy and special interest groups. With thousands of bills undergoing hundreds of changes and no nation-wide standard for publishing the information, there is a vast body of information about the future of the daily lives of Americans that is difficult to access, track and analyze.

LegiNation was founded with the goal of making state level legislation more readily available to the professionals who need it, and even more importantly to the public at large. The company builds products and websites to collect, organize, and make legislative data available to spark a renaissance in American politics. Leveraging the internet, LegiNation makes it possible to easily find, access and track proposed laws that impact their region, personal or business interests and rights.

Tackling the unstructured data challenge with practical solutions

Karen Suhaka, founder and president of LegiNation says the hard part is processing what bills are published on various government sites and published through other means and turning that information into structured data. To structure the data in a way that would be useful to people who wanted to track certain subjects but might not know what a particular bill was titled, content-specific keywords needed to be added into the metadata of the documents in their database.

Keywords needed to be extracted from large unstructured data files in four initial categories: names of people or groups involved with the bill; topics such as wildlife, education or health; the titles of laws that are to be modified; legislators who authored or introduced the bill.

Karen’s initial review of open source solutions proved unproductive. The solutions available were not flexible enough to use and could not scale to the amount of data that would need to be processed. While she could purchase enterprise level search engine tools, these would perform full text searches which would take too long to produce results and wouldn’t have the capabilities to support tracking keywords for the purposes of trending and data visualizations.

Karen found Pingar and soon began using its free developers’ sandbox account to test the API.

"Pingar was unbelievably easy to use. It required hardly any coding and no new hardware. Delivering a free sandbox version is just fantastic. We were able to ensure that Pingar's API could deliver the results we need and know we could achieve our goals before we made any investment," Karen says.

A fine sieve filters data for more useful results

Although some states publish the general subjects of bills, the categories were too broad to be useful in determining the bills that would impact people specifically. An application needed to be created to deliver a fine sieve for searching through the mountains of data available and finding the right information.

Using the Pingar API, the LegiNation team was able to create the application they needed to extract and assign content specific metadata tags to their data in just a few hours.

“Even though we were comparing Pingar to free services, Pingar is an excellent value for the money. We were able to get our project up and running in a week. That was simply not possible with other services."

Pingar’s cloud-based services helped to simplify the process, minimize risks and eliminate the need for the company to invest in additional hardware.

"As a startup, we need to get the most from our technology investment. Pingar's just-what-we-need payment model allows us to manage our data and our budget.”

Future innovation in revenue streams

The capabilities of the Pingar API will allow LegiNation to create new services and revenue streams from the data they are managing and create a publicly available online legislation tracking system.

Based on the keywords that Pingar produces, users can subscribe to a saved search  at www.BillTrack50.com that lets them know if there are changes to bills, votes, committees, or new bills that apply to their search. LegiNation has already developed one new product, a widget, where a feed on activity related to a certain topic, state or bill can be pasted into website, syndicating the content and making it more available for better decisions. Finally, the keyword data will be used to develop data visualizations to quickly summarize the fiscal or geographic impact of bills, and what bills are at what stage in the approval process.

Using this information, interest groups can make better decisions about which bills to act on and how to act.

About the Pingar API

The Pingar API works seamlessly with existing enterprise content management systems and increases the power and usefulness of your enterprise information. Pingar’s process is intuitive and yields more contextually relevant results and tools to extract information and transform that information into intelligence. The decision to release Pingar’s base technology as 18 core API components has resulted in significant market traction and is providing major opportunities for future growth across multiple domains.

About Pingar

Pingar’s mission is to transform unstructured data in the enterprise into business intelligence by developing technology that harnesses the power of text mining, natural language processing and machine learning. This mission came from years of groundbreaking research at some of the most respected universities in these fields. Pingar is headquartered in New Zealand with offices in Hong Kong, India, the United Kingdom and the United States. For more information please see www.pingar.com

Technical Specifications:

Software in use:

1. SQL Server 2008
2. .Net
3. PHP, CSS