What’s New in Node.js and libuv – June 27, 2013

Welcome to this week’s wrap up of the week in Node and libuv covering June 20 through June 26. The purpose of this blog is to recap a subset of the non-documentation related commits to Node.js, plus give a little color and commentary to the ongoing development of Node.

Node v0.10 branch highlights

string_bytes: properly detect 64bit

TJ Fontaine fixed a bug with commit a2c4ca0 that could cause ASCII scanning and/or conversion to fail on 64 bits Windows.

Revert “configure: fix cross-compilation host_arch_cc()”

Ben reverted commit acbdabb (“configure: fix cross-compilation host_arch_cc()”) in a0b6df0.  Several people reported it broke their cross-compilation workflow.

You can view the complete Nodev0.10 commit history on GitHub.

Node master branch highlights

buffer: write strings directly from call

Trevor added a new Buffer::New() method in f5e13ae that makes creating a buffer from a string a little more efficient. This will likely only be relevant to native add-on authors.

Punycode: update to v1.2.3

The Punycode module got updated to v1.2.3 in commit 24ba9fd, courtesy of Mathias Bynens.  It fixes an overly inclusive surrogate pair check: it erroneously tagged low surrogates as high surrogates. For those interested in further reading, dig the UTF-16 Wikipedia article.

You can view the complete Node master commit history on GitHub.

libuv v0.10 branch highlights

windows: use WSAGetLastError(), not errno

Ben fixed a Windows-only bug in c8c775b where setsockopt() errors weren’t being reported properly. It is unlikely to have affected anyone but he back-ported it to the v0.8 branch anyway.

You can view the complete libuv v0.10 commit history on GitHub.

libuv master branch highlights

stream: add an API to make streams do blocking writes

Henry Rawas landed a patch in 92040eb that adds an API to do blocking writes to pipes.

Blocking writes? In Node? Yep, you heard right!

See issue #3584 for the rationale. In a nutshell, having stdio that is sometimes blocking, sometimes non-blocking is a world of pain for many people causing lost messages, etc. This new API makes it possible to have stdio always be blocking.

You can view the complete libuv master commit history on GitHub.

This week’s blogs, tutorials, how-to’s and news

StrongLoop and Nodejitsu Partner to Bring Enterprise Grade Features to Node.js

Although Node.js is becoming increasingly popular with big companies building out services which benefit from its event driven, non-blocking characteristics, its adoption is sometimes stymied by a lack of advanced features and services. Specifically, private npm registries and a certified distribution for Linux and Windows, backed by technical support plans. In recognition of these needs, StrongLoop is excited to announce a strategic partnership with Nodejitsu to bring these features to the Enterprise.

Developers have made Node insanely popular

Why is the Enterprise getting excited about the potential for Node? It no doubt has to do with the enthusiasm and lobbying of developers, architects and even CTOs. Although Node has only been around for about four years, it has become the second most popular project on GitHub.

And, it doesn’t look like interest in Node is going to subside anytime soon. As of February 2013, Google Trends has been reporting that “Node.js” has overtaken “Ruby on Rails” in search volume.

Node runs Enterprise apps

Plenty of Enterprises have bet big on Node and many are hiring developers with Node specific skills to work on projects in the pipeline. Here’s a small sampling of those companies:

If you want to learn more or simply need some ammo to convince your boss that your next project should be based on Node, we’ve created an infographic that dives into the details of Node’s popularity, who’s using it and who’s supporting it.

As Enterprise adoption increases, so do the challenges

We can all agree that an Enterprise is going to have a longer set of requirements and constraints for Node, than a startup just setting out.  Fortunately, companies can now confidently deploy Node in an Enterprise setting with many of these considerations handled through offerings from StrongLoop and Nodejitsu.

StrongNode – A certified distribution of Node.js

The StrongNode distribution gives you a path forward to get technical support and bug fix coverage before heading into production. For developers who are new to Node, StrongNode gives you the perfect starting point. Navigating over 33,000 modules and the unique features of Node itself can be an overwhelming experience. StrongNode comes with vetted modules including:

  • Express – web application framework
  • Connect – rich middleware framework
  • Passport – simple, unobtrusive authentication
  • Mongoose – elegant mongodb object modeling
  • Async – higher-order functions and common patterns for asynchronous code
  • Q – a tool for making and composing asynchronous promises in JavaScript
  • Request – a simplified HTTP request client
  • Socket.IO – cross-browser WebSocket for realtime apps
  • Engine.IO – transport layer for real time data exchange
  • SL Task Emitter – perform an unknown number of async tasks recursively
  • SL Config Loader – recursively load config files
  • SL Module Loader – separate your app into modules loaded by config files
  • SL MQ – MQ API with cluster integration, implemented over various message queues

Bottom line: An organization that starts development on StrongNode ensures that by the time they go to production, they will have a company and a feature set that stands behind them.

Why an Enterprise needs a private npm registry

A few weeks ago, Nodejitsu discussed how important the npm module ecosystem was to the success of Node. But, how can an organization bring the same success in-house?

Unfortunately, this is easier to accomplish in theory than in practice. This is because the same qualities that make npm incredibly useful for developers on a small scale, can become a serious hindrance in an Enterprise development setting. Here’s a few reasons why:

  • Everyone shares the same public registry – By installing your Node.js packages from a dedicated private npm registry, you can greatly decrease your deployment times.
  • Geographic challenges – By co-locating your private npm registry with your applications, you remove any latency concerns when deploying.
  • All packages are public – If you’re not running a private npm registry today, that means that anything your developers publish to npm is public. By switching to a private npm registry you can securely install all of your packages from a single, easy to manage location.
  • Unexpected wildcards – All Node packages use [semantic versioning][semver] to specify their dependencies. The one downfall of semantic versioning is inconsistent dependencies for a single application deployed at different times. In a large enough organization you want to be sure that simple developer errors do not cause defects in your applications.
  • Managing users is painful – The Nodejitsu Enterprise npm registry can automatically replicate your developer’s npm.org authentication information. This saves you and your team from the tedious process of managing user accounts.

What’s next?

  • Ready to develop APIs in Node.js and get them connected to your data? Check out the Node.js LoopBack framework. We’ve made it easy to get started either locally or on your favorite cloud, with a simple npm install.

Comparing Node.js Support on PaaS Hosting Providers

As of May 2013

Node.js may still be young relative to its other language counterparts, but when it comes to hosting, there are a lot of options. In this post we’ll take a look at several “Platform as a Service” providers.

I’m not including “Infrastructure as a Service” options like AWS and Joyent, although the lines between PaaS and IaaS get a little blurry with some of these options.

In this round, I’m primarily looking at two aspects: deploying and configuring environment variables. I’ll also include some notes about getting started, some screenshots of dashboards, and other miscelaneous observations. In future posts we’ll run some basic performance testing and take a look at the ease of scaling.

The Players

PROVIDERDEPLOYMENTENV VARSPERFORMANCE
NodejitsuCLICLI or web interfaceSee here
HerokugitCLISee here
ModulusCLI or web uploadCLI or web interfaceSee here
AppFogCLICLI or web interfaceSee here
AzureCLI or gitCLI or web interfaceSee here
dotCloudCLICLI or .yml fileSee here
Engine Yardgitey_config npm moduleSee here
OpenShiftgitSSH and create fileSee here
CloudFoundrycoming sooncoming soonSee here

The Setup

I’m starting with a very simple Express app, and using nconf to elegantly handle the varying ways that we’ll be specifying the port our app should listen on (sometimes required) and a dummy variable I’m calling SECRET. It will first look for arguments passed in to the node command, then environment variables, then it will try to load a config.json file one level up from the root, and finally fall back on some defaults we’ve defined right in the code. When I load the app, I’ll be able to tell if our variables are being correctly pulled from one of those external sources. If it’s not, the response I get back when I load the app will be default secret. I should also be able to see what port the app is listening on and the NODE_ENV it’s using if I can access the logs from the startup of the app.

Finally, I’m setting "engines": { "node": "v0.10.x" ... in the package.json file to see how each provider reacts.

And now in no particular order…

Nodejitsu

https://www.nodejitsu.com/

One of the original players and still purely a Node.js solution, Nodejitsu became an official partner of Joyent’s back when Joyent dropped their no.de service (shame, it was such an awesome domain name). Nodejitsu no longer has a permanently free tier, but individual plans start at $9 a month and there’s a 30-day free trial.

Configuring variables

According to the docs it doesn’t matter what you set the listening port to, as long as it’s either 80 or greater than 1024.

Setting our SECRET to override the default was straightforward. You can use the CLI or web interface to list and set variables, much like several other providers on this list.

Deploying

Pushing your code to the Nodejitsu cloud is done via a custom command-line interface application (CLI), installed with npm. When you sign up you get dumped right into a github repository with instructions, but overall the setup process was pretty painless. You’re prompted to select a subdomain, which is then automatically added to the package.json file. In the couple of tests I ran, deploying was really quick. It auto-increments the version property in the package.json file with each deploy, which doesn’t bother me but might annoy some folks.

I ran into three small glitches. The first was with versioning. In the message that gets spit out upon deploying, I was shown:

info: jitsu v0.12.10-2, node v0.10.4

and yet, I was told that 0.10.x was not a supported value. Only by bumping down to 0.8.x was I able to find success.

Second, I tried to change the name property in package.json and it then wouldn’t let me deploy.

Third, my ENV variables got wiped out every time I re-deployed. Perhaps there’s a way to avoid that.

Misc Notes and Dashboard

I like that Nodejitsu is so Node.js centric. Any custom configuration is at least handled via the standard package.json file. You can even define custom predeploy and postdeploy hooks. My totally subjective response is that it felt very snappy to deploy and view logs.

Nodejitsu dashboard

Heroku

https://www.heroku.com/

The 800 pound gorilla of the PaaS world, made insanely popular by their undying love from Ruby on Rails enthusiasts everywhere.

Configuring variables

Setting our SECRET to override the default was again using the CLI. No surprises there.

All apps run on port 5000, so you need to be listening for that.

Finally, you have to create a Procfile that has specifies web: node server.js. Not a big deal, but certainly a side effect of running on a PaaS that supports multiple languages.

Deploying

The Heroku “toolbelt” CLI is used to manage your account and applications, but deploying is done via git. You just add the endpoint they provide you as a remote in your git config. Because they’re not exclusively focused on Node.js, I was pleasantly surprised to find that they already supported v 0.10.6!

My first deployment seemed to succeed, but tracking down the errors I received lead me to discover that I first needed to specify how many resources I wanted devoted to this app:

heroku ps:scale web=1

After that, it was smooth sailing.

Misc Notes and Dashboard

I didn’t try Heroku for my own projects until about 3 months ago, partly because I have a modest level of comfort in setting up my own servers, and partly because I just figured they treated node.js as an afterthought. But if you can get over their lack of WebSocket support and some misleading marketing and stats, it’s a pretty smooth experience.

They’ve also got a very polished and functional dashboard, with some handy features you won’t find elsewhere like pointing to an S3-hosted file for 404 pages, and the ability to transfer ownership of a project to a different user.

Heroku dashboard

Modulus

http://modulus.io

The .io extension should tip you off that this is a relatively new service. Focused just on Node.js, they have built-in support for MongoDB and local file storage.

Configuring variables

Variables can be set either via the web interface or the CLI. I got stuck trying to create the SECRET. After a bit of trial and error, I discovered that you can’t include spaces in the value! So modulus secret didn’t work. Weird.

The app needs to listen on port 8080, “but we recommend using the PORT environment variable (process.env.PORT).”

Deploying

Deploying can be done via CLI, but you can also zip your whole project and upload it via their web interface, which is… interesting. I didn’t have any problems deploying, but your entire project (except node_modules) gets bundled and uploaded every time, which makes it a much slower process than tools that use the “diff” capabilities of git or rsync.

As of writing, Modulus runs v 0.8.15 and ignores whatever you specify in package.json

Misc Notes and Dashboard

I’m cheering for these guys and hope they continue to improve. It’s nice having the option of built-in MongoDB, and I like the upfront pricing.

Modulus dashboard

AppFog

http://appfog.com

The PaaS formerly known as PHP Fog and recently acquired by CenturyLink. You can specify which cloud you want to run in, from several AWS regions, HP, an even Azure.

Configuring variables

Variables can be set either via the web interface or the CLI. No problems setting our SECRET variable.

The docs said I needed to listen to process.env.VCAP_APP_PORT for the port, but I just tried it with the default and it worked. The logs showed that it was listening on 57277.

Deploying

Also via the CLI. And as of writing, AppFog runs v 0.8.14 and ignores whatever you specify in package.json

Misc Notes and Dashboard

The free plan seems pretty generous, giving you up to 8 instances across 2GB of RAM, split however you want. They also have options for built-in MongoDB.

AppFog dashboard

Windows Azure

http://windowsazure.com

Yep. Believe it.

Configuring variables

Variables can be set either via the web interface or the CLI. I experienced one weird bit behavior on the SECRET when I first set it up via the CLI; it was lowercased to secret. I had to use the web interface to correct it.

The app ran fine without making any changes to our configuration for ports, although the logs showed a pretty wacky value: \.pipebea0dffc-de5b-47f4-8575-4d17c2175fd5

Deploying

You might expect Microsoft to have all kinds of custom stuff here. While there are a few files that get dropped into your repo (name azure_error and iisnode.yml), the (optional) CLI is installed via npm and deployment is handled via git. Nice!

As of writing, Azure is running 0.8.2, and this value must be configured correctly in package.json.

Misc Notes and Dashboard

The account creation process was by far the most tedious of any listed here. It included SMS verification, a Windows Live account, and a separate git user account for deploying. But overall, the dashboard is pretty nice considering the huge scope of the Azure platform. They also support streaming log files. As far as I can tell, Heroku is the only other provider on this list to do so.

Azure dashboard

dotCloud

http://dotcloud.com

Support for multiple languages. Like Nodejitsu they recently dropped their free tier, so a credit card is required to get started.

Configuring variables

Variables via the CLI or a dotcloud.yml file.

The listening port needs to be set to 8080.

Deploying

dotClouds CLI is written in Python and uses rsync to minimize the amount of data you need to upload on each deploy. In order to deploy, you need both a supervisord.conf and dotcloud.yml files, which is a minor nuisance.

As of writing, dotCloud runs 0.6.20. Ouch.

Misc Notes and Dashboard

I got burned early on by dotcloud when they made some breaking changes with little warning. I guess that’s life in a beta world, but I haven’t been anxious to run back.

dotCloud dashboard

Engine Yard

http://engineyard.com

Like Heroku, Engine Yard is known primary for it’s Ruby on Rails hosting. Unlike Heroku, you have more control over the setup and environment. It’s still a PaaS, but people with some direct server experience will appreciate being a little “closer to the metal.”

Configuring variables

Environment variable cas be set by using the ey_config npm module. Engine Yard also support SSH access to your machine, which means you could create a config.json and drop it in the right place.

The port should be process.env.PORT and nconf is already handling that for us.

Deploying

Handled via git.

It respects the node version value in package.json but only up to 0.8.11.

Misc Notes and Dashboard

When you create an application, you get to choose between nginx as the front-end proxy (which the recommend) or going straight to Node.js, the latter of which gives you option of WebSockets.

When my first deploy failed, a chat window popped up with option for me to put in my phone number.

Engine Yard dashboard

OpenShift.com

http://openshift.com

OpenShift is an open source project maintained primarily by Red Hat, and OpenShift.com is a hosted service for the platform (think WordPress.org vs WordPress.com).

Configuring variables

OpenShift.com supports SSH access to your machine, which means you could create a config.json and drop it in the right place. Otherwise, there are no special commands or predefined config files.

The port should be process.env.OPENSHIFT_INTERNAL_PORT || 8080.

Deploying

Handled via git, though their is a CLI for creating and managing projects if you’re not a fan of web-based GUIs. You’ll need to add your public SSH key to your account first, which was a step not included in the node.js sample quickstart.

It ignores the node version value in package.json and runs a rusty ol’ v0.6.20 by default, but with some extra steps you can change that.

Misc Notes and Dashboard

I had some problems getting the sample app running at first, so I started over and let OpenShift populate the git repository for me with their sample files. There are a few extra things in there, many of which are tucked under the.openshift directory. These include deploy hooks and config files. The README they include does a good job of laying out the structure and explaining the purpose of each file.

But I still got stuck on the fact that I needed to listen on a specific IP address. That meant including this:

var ipaddr = process.env.OPENSHIFT_NODEJS_IP || "127.0.0.1";

And then I added ipaddr in the app.listen section of my server.js like this:

app.listen(app.get('port'), ipaddr, function(){...

I’ve had an exchange with the OpenShift folks and they’re going to update their getting started guide to make this more clear.

OpenShift dashboard

CloudFoundry

http://cloudfoundry.com

At first look, CloudFoundry has a clear and friendly process for getting started. However, a few screens into the process I was presented with this message:
*We are in a time of transition:*
You are reading V2 documentation... Cloudfoundry.com will be upgraded to V2 in June 2013.

So I’ll circle back to them in the next installment of this series.

Until Next Time

Needless to say things are changing quickly in this space. Don’t put too much weight on any specific detail here without first checking for updates. For example, in the couple of days that it took me to type this up, Modulus sent out an email about upgrading their version of Node.js.

In future posts I’ll be walking through how we can monitor and test the performance of these platforms, and then take a shot at scaling them. In the meantime, let me know if I’ve missed your favorite Node.js PaaS.

 

 

What’s New in Node.js and libuv – June 20, 2013

Welcome to this week’s wrap up of the week in Node and libuv covering June 13 through June 19. The purpose of this blog is to recap a subset of the non-documentation related commits to Node.js, plus give a little color and commentary to the ongoing development of Node.

Node v0.10 branch highlights

Node v0.10.12 (Stable) and v0.8.25 (Maintenance) are out

v0.10.12 and v0.8.25 are two new releases of v0.10 which was announced in the last week, both of which included upgrades to npm. The maintenance release contains a back-port that fixes handle delivery to child process in commit 8a3d0c8.

Read the official blogs for v0.10.12 and v0.8.25 for the complete scoop.

Revert “http: remove bodyHead from ‘upgrade’ events”

Isaac pushed commit e850027 which reverted Nathan Zadoks change to the http module that was breaking socket.io.

openssl: add missing configuration pieces for MIPS
v8: fix build for mips32r2 architecture

Andrei Sedoi fixed the MIPS build in 884b253 and cc51749. The second commit being a back-port of an upstream V8 commit.

net: Do not destroy socket mid-write

Isaac fixed some fallout from e0519ac in 3c7945b where a net.Socket object was sometimes destroyed prematurely.

v8: add setVariableValue debugger command

Ben backported upstream V8 debugger functionality in 41fc46e that lets you modify local variables in the debugger. This should make node-inspector a little more useful in v0.10.

readline: make ctrl + L clear the screen

Yuan Chuan added Ctrl+L (clear screen) support to the REPL and readline module in 18574bf. Ben points out that he personally likes this feature alot, “I’m a compulsive ^L-er.”

You can view the complete Node v0.10 commit history on GitHub.

Node master branch highlights

process: abstract out HandleToStream
stream_wrap: introduce StreamWrapCallbacks
tls_wrap: embed TLS encryption into streamwrap
tls: introduce TLSSocket based on tls_wrap binding
tls: share socket._hadError with http_client

Fedor landed a new TLS implementation in commits 6978e99, 4c48a39, 03e008d, af80e7b, dc50f27 and 212e9cd that is easily twice as fast as the one in v0.10.

readline: strip ctrl chars for prompt width calc

Krzysztof Chrapka offered up commit ffcd8b9 in which embedded control characters no longer throw off height and width calculations in the readline module. This fixes bugs #3860 and #5628.

build: add android support

It became a lot easier to build Node.js on (or for) Android with commit 5e4e8ec by Linus Mårtensson.

buffer: proper API export for Windows
buffer: implement new fill behavior
buffer: deprecate legacy code
buffer: remove c-style casts
buffer: expose class methods alloc and dispose
buffer: reimplement Buffer pools
buffer: use smalloc as backing data store
smalloc: add api to manually dispose Persistent
smalloc: initial implementation

Trevor put together a new Buffer API in commits 8f3f9f7 through 7373c4d that makes buffers considerably more versatile and efficient.

You can view the complete Node master commit history on GitHub.

libuv v0.10 branch highlights

linux: add support for MIPS

Andrei added MIPS support in 5096f1e.

windows: uv_spawn shouldn’t reject reparse points

Bert pushed 399c3ef which fixed a bug in the uv_spawn() code where it refused to follow reparse points, the Windows equivalent of symlinks. In other words, it wasn’t possible to run a symlinked binary. Curiously, Windows developers probably don’t use symlinks much because this bug has gone unnoticed for a long time. It has rather amazingly been in there since the beginning of Node.

You can view the complete libuv v0.10 commit history on GitHub.

libuv master branch highlights

Just minor fixes this go around, check back next week.

You can view the complete libuv master commit history on GitHub.

This week’s blogs, tutorials, how-to’s and news

Announcing StrongNode 1.0.2 GA – Maintenance Release

StrongNode 1.0.2 Maintenance Release is now available for Windows, Linux and MacOS X. It comes packaged with the latest and greatest, Node.js v0.10.11 (Stable). In case you missed the official announcement blog last week, new in this release:

  • npm upgraded to 1.2.30
  • libuv upgraded v0.10.11
  • plus more…

If you’d like to take a deeper dive into what’s new beyond the official release blog, you should check out our weekly “What’s New in Node.js and libuv” blog posts which give you plenty of color and commentary on what’s happening on the development front.

Of course, StrongNode still ships with the slc command-line utility and a set of supported npm modules, including:

  • Express – web application framework
  • Connect – rich middleware framework
  • Passport – simple, unobtrusive authentication
  • Mongoose – elegant mongodb object modeling
  • Async – higher-order functions and common patterns for asynchronous code
  • Q – a tool for making and composing asynchronous promises in JavaScript
  • Request – a simplified HTTP request client
  • Socket.IO – cross-browser WebSocket for realtime apps
  • Engine.IO – transport layer for real time data exchange
  • SL Task Emitter – perform an unknown number of async tasks recursively
  • SL Config Loader – recursively load config files
  • SL Module Loader – separate your app into modules loaded by config files
  • SL MQ – MQ API with cluster integration, implemented over various message queues

What’s next?

  • Ready to develop APIs in Node.js and get them connected to your data? Check out the Node.js LoopBack framework. We’ve made it easy to get started either locally or on your favorite cloud, with a simple npm install.