If You Can Read This, You're SNIing
Friday, 9 May 2014
When TLS was defined, it didn’t allow more than one hostname to be available on a single IP address / port pair, leading to “virtual hosting” issues; each Web site (for example) now requires a dedicated IP address.
HTTP solved this problem with the Host request header; TLS did it with the Server Name Indication (SNI) extension. You can read more about it at the Wikipedia page.
The problem is that some clients – most notably Windows XP of any flavour, Android 2.x, and Java up until 1.7 – don’t send SNI information. Since requiring SNI effectively orphans these clients, people have been reluctant to require it on their Web sites, even though it saves precious, precious IPv4 addresses.
An Experiment
However, I believe in living on the bleeding edge, want to get more deployment experience with the protocols we’re developing, and most importantly, don’t want to give Rackspace the extra $2 a month for another IP address.
So, both this site and redbot.org have switched to HTTPS URIs, and require the use of SNI to access them. My audience tends not to run Windows XP, so I think that’s a reasonable thing to do, and somebody has to take the first step.
(BTW - for those who think this site may have got a bit slower, it’s not because of the TLS – it’s because I moved the server from Texas to Sydney.)
Along the way, I tried to get reasonably close to best practice for both security (as outlined by bettercrypto.org) and performance (see Is TLS Fast Yet?). Once things settle down a bit more, I’ll turn on Strict Transport Security too.
The SNI configuration for Apache 2.4 was a bit fiddly, mostly because I didn’t want to set a “default” site for clients that don’t send SNI; I wanted a hard fail. The relevant config ended up looking like this:
SSLStrictSNIVHostCheck on
ErrorDocument 403 "TLS SNI Required."
Listen 443
<VirtualHost *:443>
...
SSLStrictSNIVHostCheck on
<Directory ...>
ErrorDocument 403 default
SSLRequireSSL
SSLOptions +StrictRequire
</Directory>
</VirtualHost>
In my testing, this generates a 403 with the message “TLS SNI Required.” for clients that don’t send SNI information.
403 really isn’t the greatest status code for this semantic, so I filed a bug with Apache to make it more appropriate, and to give more flexibility in the error document (since most people’s reaction to that message will be, deservedly, “huh?”).
Stories from the Logs
Interested to see what the effect of requiring SNI would be, I set my server up to log the server name presented:
LogFormat "%h %t %D %u %{SSL_TLS_SNI}e \"%r\" %>s %R %b %X %I %O \"%{Referer}i\" \"%{User-agent}i\"" site
Clients that don’t present SNI get logged as “-“, and of course get a 403.
After a bit of command-line grep / sort / uniq foo, here is a list of UAs that didn’t present SNI information. In some cases, I’ve trimmed the “helpful” compatibility information out (e.g., Mozilla/blah…). I’ve also taken out the browsers that don’t support SNI (almost all on XP or older versions of Windows).
Sites, Services
These are online sites and services that made fetches without SNI, where I can’t determine what their implementation language is.
- CareerBot/1.1; +http://www.career-x.de/bot.html
- FeedBooster; +http://feeds.qsensei.com
- FlipboardProxy/1.1; +http://flipboard.com/browserproxy
- FlipboardRSS/1.1; +http://flipboard.com/browserproxy
- Hatena Antenna/0.5 (http://a.hatena.ne.jp/help)
- Hatena::Bookmark/2.00
- James BOT - WebCrawler http://cognitiveseo.com/bot.html
- kouio.com RSS reader - 5 subscribers
- Kraken/0.1; http://linkfluence.net/; bot@linkfluence.net
- livedoor FeedFetcher/0.01 (http://reader.livedoor.com/; 26 subscribers)
- LiveJournal.com (webmaster@livejournal.com; for http://www.livejournal.com/users/mnot_main/; 2 readers)
- NewsBlur Feed Fetcher - 263 subscribers - http://www.newsblur.com/site/1139/mnots-blog
- PercolateCrawler/4 (ops@percolate.com)
- pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm
- PushBot/1.1; +support@appnotifications.com
- TBRSS/1 (https://tbrss.com/; 1 subscriber)
- woriobot +http://worio.com
Seach Engines
Really just a sub-category of Sites and Services, search engines that didn’t send SNI include:
- Baiduspider/2.0; +http://www.baidu.com/search/spider.html
- BazQux/2.4; +https://bazqux.com/fetcher; 9 subscribers
- BlogSearch/2 +http://www.icerocket.com/
- bingbot/2.0;+http://www.bing.com/bingbot.htm
- Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot
- EasouSpider; +http://www.easou.com/search/spider.html
- Exabot/3.0; +http://www.exabot.com/go/robot
- MSIE or Firefox mutant; not on Windows server; + http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0
- MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+
- msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
- msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)
- Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp
- YandexBlogs/0.99; robot; B; +http://yandex.com/bots)0 readers
- YandexBot/3.0; +http://yandex.com/bots
- YandexImages/3.0; +http://yandex.com/bots
Tools
These ones look like desktop or mobile tools, but I can’t tell the implementation language.
- Feedreader 3.14 (Powered by Newsbrain)
- HTTrack Website Copier/3.0x (offline browser; web mirror utility)
- Instapaper for Android 3.0
- Pogo Reader (feed reader)
- RSSOwl/2.2.1.201312301258 (X11; U; en)
- Seznam screenshot-generator 2.0; +http://fulltext.sblog.cz/screenshot/
Unknown
Not sure about these. shrug
- Gregarius/0.6.0 (+http://devlog.gregarius.net/docs/ua)
- http_get
- kjb-RSS/5.13
- MetaURI API/2.0 +metauri.com
- NerdyBot
- newspaper/0.0.6
- news_publisher_rss_spider/1.0
- rawdog/2.19
- RTWT http://rtwt.aloodo.com/
- RubeGoldbergPostalService/1.0
- seo/1.0
- u12Bot/5.0; +http://u12files.com/
Language-Specific
Finally, agents that can be tracked to a specific implementation language.
Java
Java supports SNI since 1.7, but of course there are still some old clients out there:
- CommaFeed/1.0 (http://www.commafeed.com)
- JDK/1.0.2 Java/1.6.0
Perl
Apparently, Perl IO::Socket::SSL has supported SNI for a while now, provided that the underlying OpenSSL does. These people should just need to upgrade OpenSSL and rebuild their Perl.
- libwww-perl/5.834
- W3C_Validator/1.3 http://validator.w3.org/services
PHP
PHP gave control over SNI in 5.3.2, but if they’re using Curl, it’s just a matter of getting the right versions.
- MagpieRSS/0.72 (+http://magpierss.sf.net) (Tiny Tiny RSS/1.4.1)
- PECL::HTTP/1.7.1 (PHP/5.3.10)
Ruby
It looks like Ruby has supported SNI for a while now. Not sure what’s going on here.
- EventMachine HttpClient
- Feed2Imap v1.0 http://home.gna.org/feed2imap/
Python
Later releases of Python 3 have SNI, and Python 2 should be getting it in the next few weeks, when 2.7.7 is released. Hopefully these folks will upgrade and this list will thin down soon:
- bookie / (https://github.com/bookieio/bookie)
- Feedjack 13.08.16 - https://github.com/mk-fg/feedjack
- FeedValidator/1.3
- LinkChecker/8.4; +http://wummel.github.com/linkchecker/
- plagg/3.0 (+http://drbeat.li/py/plagg/)
- pnntprss/0.01 +http://david.wragg.org/pnntprss/
- Python-httplib2/$Rev$
- python-requests/2.2.1 CPython/2.7.6 Darwin/13.1.0
- Python-urllib/2.7
- RED/1 (http://redbot.org/)
- rss2email/2.71 +http://www.allthingsrss.com/rss2email/
- UniversalFeedParser/5.1.3 +https://code.google.com/p/feedparser/
Conclusions
It’s early days, of course, but this makes me pretty hopeful; if Python gets fixed soon, it knocks out a fair bit of the list, and there’s a small-ish list of sites and services that needs to upgrade (which is dwarfed by the list of those that do support SNI, btw).
2 Comments
openid.seld.be said:
Saturday, June 7 2014 at 9:09 AM
yuhong.wordpress.com said:
Monday, June 30 2014 at 4:50 AM