You wanted it? Here it is.

Creating a Mint.com API

Jun
29
Published in: Web Development with 12 comments.

I have recently started using Mint.com for my personal finances. At first, I was very skeptical about the security of things at mint. I was unsure whether or not I could trust the connection and my personal banking information to be accessed by another person and/or company. In short, after reviewing their security policy and taking a closer look, I was able to move past my insecurities.

With ease, I have been able to connect all of my banking information in just a few simple clicks… Pretty scary, huh? I think this is an amazing feature for banking 2.0 sites. Mint.com takes care of organizing your finances through an easy-to-use interface. All this is great but, as a developer, my mind started thinking about how I can make use of this site. And so, the task of investigating whether or not mint.com has an official API to pull transactions, account balances, and other things of the like. It would be great if I could integrate mint.com support into different applications and backend’s that I write.

After a couple hours of perusing through the Internet’s web of data, I was able to find a large amount of user requests for exactly this sort of thing. Seems like I’m not the only one to have had this idea, nothing new here… While there are definitely pro’s and con’s of having an API to access account information, I think that this feature would prove very beneficial to the site. There are a multitude of things I could use mint.com API support for. Anyways, without deviating from my original point… Wait, what’s that? … You guessed it! I began to play with the idea of creating a high-level PHP class to support scraping mint.com for user information.

Within around an hour or so, I was able to automate the mint.com login and scrape all of my accounts, including:

  • Account Name
  • Time last checked
  • Account Balance
  • Account Nickname

It was actually rather easy to do. It’s a simple case of using PHP and cURL to fetch the information you need. It looks a little something like this:

$url = 'https://wwws.mint.com/loginUserSubmit.xevent';
$post = array(
	'username' => $username,
	'password' => $password,
	'task' => 'L'
);

//build the query _GET string
$post = http_build_query($post);
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 12);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt');

//get the results now
$result = curl_exec($ch);
$error = curl_error($ch);
curl_close($ch);

That bit of code sets up a cURL object that contains all of the appropriate settings/POST information that it needs to log into the mint.com site. This cURL request requires a cookie jar to store the result of the mint.com login script(s). After you have the result, you can actually begin scraping out the individual accounts by exploding on the text “</a></h4><h6><span>” and filtering through each match to make sure that you can pull the information that you want.

Now that we’ve generally covered the scraping process, you can take it another level by performing things like:

  • Retrieving account transactions
  • Performing a refresh account request
  • Getting account alerts
  • Pulling budget information
  • Getting mint.com charts
  • And all of the other great stuff Mint.com has to offer…

In the hour I had to experiment with this idea, I was able to toy with the idea of performing a “Refresh Account” request. This request, along with the others i just mentioned in the list above, require a little more work. I still have not perfected the request due to lack of time. But from what it looks like, mint.com initiates a server-side request to refresh all accounts attached to the current user. A token is passed to the refresh script as another layer of security. After reviewing the source of the overview page, it looks like the token can be directly pulled from the html, so you would need to simply extrapolate this data in order to perform tasks. I have tested this by just posting the extracted token to the url, along with the original cookie jar, and I receive a response: “Session has expired.” This makes me believe that mint.com is expecting information from cookies that have been generated client-side. The next step is to basically pull the data that is generated client-side and post it to the url above, along with the token and the cookie jar. The token can be found in a hidden input with the id “javascript-user”. I think the naming convention confirms my thought process about client-side cookie generation. I will need more time to look into this further and have plans to post my findings when I get to them.

The reason the refresh is so important is because of the slow HTTPS connection that occurs when you log in with cURL and PHP. To speed this up, we could auto-cache a first result of the accounts and their balances, much like what mint.com does. After the accounts have been cached, we can pull the data from MySQL almost instantly and then give an “Update All” button to the user just like mint.com. This would help us achieve faster page load speeds as we won’t be waiting for a 10 second script to finish loading dynamically.

I say all of this to mention that getting mint.com account data isn’t all that bad… However, I’d prefer an official REST API or something of this sort.

Reply
Ethan on July 17, 2011 at 5:16am wrote:
I am playing with the same thing and using your posted code, but am not able to log in successfully. I am bounced to a page that begins "You have successfully signed out of TurboTax".

I am only trying to get at my transactions, which should be accessible via this URL once I am authenticated:

https://wwws.mint.com/transactionDownload.event?accountId=5077467 (for each account)

I am using PHP 5.2.6 from Debian Lenny, with Curl 7.18.2
Reply
Ethan on July 17, 2011 at 5:40am wrote:
Nevermind, I somehow had a problem with my auth information. It is working like a charm, and the transaction download is what I expected as well. If you request https://wwws.mint.com/transactionDownload.event you get all transactions (there is an Account Name column, so you can split them out if you need to) and if you append ?accountId=5077467 to the request (using your account number) then you'll get back transaction for only that account. This is in CSV with a descriptive header line. Next step is pushing the data to my custom Google spreadsheet budget.
Reply
Nick on July 17, 2011 at 5:47pm wrote:
You're definitely on to something here. I've looked at scraping before, and it seems like a lot of work that can be easily broken by website changes. Things to consider...

1) the Mechanize library
2) beautifulSoup

those two libraries are great for manipulating data via scripting like Ruby or Python...

Also, there is a CSV export option in Mint. Could this server-side login-method be used to create an always-available CSV source link so that we can constantly pull it into apps that support web connect?

From there, you could put it into a Yahoo! Pipe and manipulate the data even further, with output formats like JSON, XML, or RSS
Reply
Carl Danley on July 18, 2011 at 3:13pm wrote:
I know that I've seen a solution for getting the data via JSON format. You could then use this data to build a csv very easily by using an old PHP trick like this:

$data = json_decode(json_encode($data), true);


At this point, you'd have a PHP array full of all the data you need. You could loop through and build a CSV file whenever you want. I'd just be interested in initiating some sort of refresh call to their site via a script. When I get some more time, I'm going to look into this a little more.
Reply
Dan on July 18, 2011 at 12:40pm wrote:
I've been "scraping" Mint's weekly account summary emails for about a year now. It's not as powerful as scraping their site, but it is stabler and simpler. If you're just trying to get account balances, it works great.
Reply
Ethan on July 21, 2011 at 1:31am wrote:
Once you are logged in using Carl's code, you can snag the CSV like I do in this function (passing a Mint account ID if you want only one account's transactions):

//retrieves CSV file of transactions
function mint_get_transactions_csv($foreign_account_id=false)
{
  //set the parameters
  $url = 'https://wwws.mint.com/transactionDownload.event';
  if ($foreign_account_id) $url .= '?accountId=' . $foreign_account_id;

  //build the request
  $ch = curl_init($url);
  curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  curl_setopt($ch, CURLOPT_TIMEOUT, 12);
  curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
  curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
  curl_setopt($ch, CURLOPT_COOKIEJAR, '/tmp/curl_cookies.txt');
  curl_setopt($ch, CURLOPT_COOKIEFILE, '/tmp/curl_cookies.txt');

  //get the response
  $response = curl_exec($ch);
  $error = curl_error($ch);
  curl_close($ch);
  if ($response) return $response;
  else return false;
}


Turning the CSV into an array is easy, just explode it a couple of times:

if ($csv)
{
  $transactions = array();
  $csv_lines = explode("\n", $csv); //convert file into an array of lines
  unset($csv_lines[0]); //remove the header line
  foreach ($csv_lines as $line)
  {
    //munge the incoming data into some semblance of order
    $line = substr($line, 1, strlen($line)-2); //strip off first and last quotes
    $line = explode('","', $line); //convert line into an array of fields
    /* .... your custom stuff here .... */
  }


Beware that Mint returns one line of the file that doesn't explode properly - it's probably just a trailing hard return at the end. I'm ignoring several of Mint's fields so further code wouldn't be generally useful, but the next step would be converting the array into an associative array for easy use. There's probably a nifty way to do this with the contents of that first line that I simply delete. It's too bad that Mint doesn't provide unique ID's, because that is the only way to discern between altered lines and new lines, for certain.

I am putting the transactions into my own database at this point, so Mint is simply providing me with a highway to all of my financial institutions' data. Next I am going to push new transactions to an RSS feed (accessible in Google Reader and from my phone) where I will see and categorize them, and from there into a Google spreadsheet for consumption in whatever form I care to create. Win! One of the most important reasons was to be able to use my mere 12 categories instead of Mint's 80+, which cannot be made convenient on a phone platform by any UI trick known to man. Why Mint doesn't allow hiding categories itself, doesn't push to RSS/phone for categorization, doesn't mark reviewed vs. un-reviewed transactions, and doesn't allow historical/manual transaction insertion remains a mystery. But now I don't have to care.
Reply
Carl Danley on July 21, 2011 at 11:52am wrote:
Great post Ethan! I'm glad you found a way to pull the data from Mint.com. This will be a very nice feature for people who need to integrate their financial information into their own site. We should really create some sort of bundled API for people. Then, people could plugin their information and "let it rip!" lol
Reply
Ethan on July 21, 2011 at 2:03am wrote:
Carl, I don't know why the "code lang=php" tag didn't work well on my submission, but here is a link to the bare code that I shared if you have a way to clean up my post and are so inclined: http://www.ethanpooley.net/mint.txt

I'd try again but I'm not sure what went wrong and don't want to spam you.
Reply
Carl Danley on July 21, 2011 at 11:23am wrote:
I believe I have fixed the problem that was happening with the code excerpts. Please let me know if something else comes up and doesn't work correctly. Still trying to finish site when I get free time :)
Reply
Ethan on July 22, 2011 at 12:00pm wrote:
Thanks for the cleanup, Carl. It does look like the contents of the $url variable are still altered - it should just be the straight url, not a whole anchor tag.

Here's the mobile-friendly page I'm using for handling unreviewed transactions: http://www.ethanpooley.net/bnfo.jpg

Easy to put in a desktop web-widget also.
Reply
Stephen on August 13, 2011 at 5:48pm wrote:
Ethan, I can't seem to get your code to work with Carl's. I put yours right after Carl's (minus the "curl_close" line), and then called the mint_get_transactions_csv function right after. I keep getting Mint's "Perhaps you took a wrong turn?" not-logged-in message. Thoughts? Could you post a more complete example?
Reply
Carl Danley on August 15, 2011 at 1:21pm wrote:
Stephen, are cookies being correctly written to your cookie file? If not, check the permissions on the cookie file and/or your script. Let me know if you still need help.

What do you think? Let me know.

* Bolded fields are required.

Name
Email (won't be posted)
Website
Supported CB Tags are:
[code lang="php"][/code], [code lang="js"][/code], [code lang="css"][/code], [code lang="html"][/code], [b][/b]