Just thinking out loud this morning on what data formats there are and how they work with the world:

• XML: 2000's hippest technology. Combines a clear, parsable tree based syntax with extension mechanisms and a schema system. Still moderately popular, though not as it once was. Tons of tooling. Many seem to think the tooling makes it overly complex, and JSON has taken over much of its place. Has the advantage of unambiguity over vanilla JSON, if you know how to use it right, but more effort to work with.
• SGML: XML's soupier grandmother. Influential.
• HTML: Kind of like SGML and XML but for some specific data. Too bad XHTML never fulfilled its dream. Without XHTML, it's even soupier than SGML, but there's enough tooling for soup-processing that most developers don't worry about it.
• JSON: Also tree-based, but keeps things minimal, just your basic types. Loved by web developers everywhere. Also ambiguous since on its own, it's schema-free... this may lead to conflicts between applications. But if you know the source and the destination perfectly it's fine. Has the advantage of transforming into basic types in pretty much every language and widespread tooling. (Don't be evil about being evil, though? #vaguejokes) If you want to send JSON between a lot of locations and want to be unambiguous in your meaning, or if you want more than just the basic types provided, you're going to need something more... we'll come to that in a bit.
• S-expressions: the language of lisp, and lispers claim you can represent anything as s-expressions, which is true, but also that's kind of ambiguous on its own. Capable also of representing code just as well, which is why lispers claim benefits of symmetry and "code that can write code". However, serializing "pure data" is also perfectly possible with s-expressions. So many variations between languages though... it's more of a "generalized family" or even better, a pattern, of data (and code) formats. Some damn cool representations of some of these other formats via sexps. Some people get scared away by all the parens, though, which is too bad, because (though this strays into code + data, not just data) homoiconicity can't be beat. (Maybe Wisp can help there?)
• Canonical s-expressions: S-expressions, with a canonical representation... cool! Most developers don't know about it, but was designed for public key cryptography usage, and still actively used there (libgcrypt uses canonical s-expressions under the hood, for instance). No schema system, and actually pretty much just lists and binary strings, but the binary strings can be marked with "display hints" so systems can know how to unpack the data into appropriate types.
• RDF and friends: The "unicode" of graph-oriented data. Not a serialization itself, but a specification on the conceptual modeling of data, and you'll hear "linked data" people talking about it a lot. A graph of "subject, predicate, object" triples. Pretty cool once you learn what it is, though the introductory material is really overwhelming. (Also, good luck representing ordered lists). However, there is no one serialization of RDF, which leads to much confusion among many developers (including myself, while being explained to the contrary, for a long time). For example, rdf/xml looks like XML, but woe be upon ye who uses XML tooling upon it. So, deserialzie to RDF, then deal with RDF in RDF land, then serialize again... that's the way to go with RDF. Has more sane formats than just rdf/xml, for example Turtle is easy to read. RDF community seems to get mad when you want to interpret data as anything other than RDF, which can be very off-putting, though the goal of a "platonic form" of data is highly admirable. That said, graph based tooling is definitely harder for most developers to work with than tree-based tooling, but hopefully "the jQuery of RDF" library will become available some day, and things will be easier. Interesting stuff to learn, anyway!
• json-ld: A "linked data format", technically can transform itself into RDF, but unlike other forms of RDF syntax, can often be parsed just on its own as simple JSON. So, say you want to have JSON and keep things easy for most of your users who just use their favorite interpreted language to extract key value pairs from your API. Okay, no problem for them! But suddenly you're also consuming JSON from multiple origins, and one of them uses "run" to say "run a mile" whereas your system uses "run" to mean "run a program". How do you tell these apart? With json-ld you can "expand" a JSON representation with supplied context to an unambiguous form, and you can "compact" it down again to the terms you know and understand in your system, leaving out those you don't. No more executing a program for a mile!
• Microformats and RDFa: Two communities which are notoriously and exasperatingly at odds with each other for over a decade, so why do I link them together? Well, both of these take the same approach of embedding data in HTML. Great when you have HTML for your data to go with, though not all data needs an HTML wrapper. But it's good to be able to extract it! RDFa simply extracts to RDF, which we've discussed plenty; Microformats extracts to its own thing. Frequent form of contention between these groups is about vocabulary, and how to represent vocabulary. RDFa people like their vocabulary to have canonical URIs for each term (well, that's an RDF thing, so not surprising), Microformats people like to document everything in a wiki. Arguments about extensibility is a frequent topic... if you want to get into that, see Amy Guy's summary of things.

Of course, there's more data formats than that. Heck, even on top of these data formats there's a lot more out there (these days I spend a lot of time working on ActivityStreams 2.0 related tooling, which is just JSON with a specific structure, until you want to get fancier, add extensions, or jump into linked data land, in which case you can process it as json-ld). And maybe you'd also find stuff like Cap'n Proto or Protocol Buffers to be interesting. But the above are the formats that, today, I think are generally most interesting or impactful upon my day to day work. I hope this guide was interesting to you!

An animated guide to all the different ways of making coffee

Enlarge / A screenshot showing proof-of-concept exploit code working against Kaspersky antivirus software. (credit: Tavis Ormandy)

Antivirus applications and other security software are supposed to make users more secure, but a growing body of research shows that in some cases, they can open people to hacks they otherwise wouldn't be vulnerable to.

The latest example is antivirus and security software from Kaspersky Lab. Tavis Ormandy, a member of Google's Project Zero vulnerability research team, recently analyzed the widely used programs and quickly found a raft of easy-to-exploit bugs that made it possible to remotely execute malicious code on the underlying computers. Kaspersky has already fixed many of the bugs and is in the process of repairing the remaining ones. In a blog post published Tuesday, he said it's likely he's not the only one to know of such game-over vulnerabilities.

"We have strong evidence that an active black market trade in antivirus exploits exists," he wrote, referring to recent revelations that hacked exploit seller Hacking Team sold weaponized attacks targeting antivirus software from Eset.

He continued: "Research shows that it’s an easily accessible attack surface that dramatically increases exposure to targeted attacks. For this reason, the vendors of security products have a responsibility to uphold the highest secure development standards possible to minimise the potential for harm caused by their software. Ignoring the question of efficacy, attempting to reduce one’s exposure to opportunistic malware should not result in an increased exposure to targeted attacks."

As Ormandy suggested, the bugs he found in Kaspersky products would most likely be exploited in highly targeted attacks, such as those the National Security Agency might carry out against a terrorism suspect or spies pursuing an espionage campaign might carry out against the CEO of a large corporation. That means most people are probably better off running antivirus software than foregoing it, at least if their computers run Windows. Still, the results are concerning because they show that the very software we rely on to keep us safe in many cases makes us more vulnerable.

Kaspersky isn't the only security software provider to introduce bugs in their products. Earlier this month, security researcher Kristian Erik Hermansen reported finding four vulnerabilities in the core product marketed by security firm FireEye. One of them made it possible for attackers to retrieve sensitive password data stored on the server running the program. Ormandy has also uncovered serious vulnerabilities in AV software from Sophos and Eset.

In a statement, Kaspersky Lab officials wrote, "We would like to assure all our clients and customers that vulnerabilities publicly disclosed in a blogpost by Google Project Zero researcher, Mr. Tavis Ormandy, have already been fixed in all affected Kaspersky Lab products and solutions. Our specialists have no evidence that these vulnerabilities have been exploited in the wild."

The statement went on to say that Kaspersky Lab developers are making architectural changes to their products that will let them better resist exploit attempts. One change included the implementation of stack buffer overflow protection, which Ormandy referred to as "/GS" in his blog post. Other planned changes include the expansion of mitigations such as address space layout randomization and data execution prevention (for much more on these security measures see How security flaws work: The buffer overflow by Ars Technology Editor Peter Bright). Ormandy thanked Kaspersky Lab for its "record breaking response times" following his report.

Still, the message is clear. To perform, security software must acquire highly privileged access to the computers they protect, and all too often this sensitive position can be abused. Ormandy recommended that AV developers build security sandboxes into their products that isolate downloaded files from core parts of the computer operating system.

"The chromium sandbox is open source and used in multiple major products," he wrote. "Don't wait for the network worm that targets your product, or for targeted attacks against your users, add sandboxing to your development roadmap today."

Read Comments