Just yesterday I was opining in the IRC about what it would be like to have a operating system that was build from the ground up that supports smarter data formats than just text.
The concept of everything as a file, has awesome benefits I believe, but that doesn't meant files need to be unstructured text.
There's a core problem, that with text languages (like AWK or structured regex (sed, etc) you end up both having to parse AND manipulate data. which is no fun and prone to errors.
Abstracting away all of that into codecs that all of coreutils could speak would be very cool to me.
The second issue is structured data vs unstructured text. CSVs, or other table based formats make sense sometimes, and sometimes you want to be able to query things more easily. JSON provides that. What's the synthesis of both I don't know... but maybe this is closer to heaven.
I'd like to _minimize_ the amount of parsing or text manipulation done in any program and be able to focus on extracting and inserting the data I need, like a rosetta stone that just handles everything for me while I work. I want to be able to do:
awk "{print $schoolName}" or awk "{print $4}" and it just works - json, text, CSV.
Maybe the most notable system that had "files as structured data" is the IBM S/360 lineage of mainframes.
> In the context of IBM mainframe computers in the S/360 line, a data set (IBM preferred) or dataset is a computer file having a record organization.
> Data sets are not unstructured streams of bytes, but rather are organized in various logical record[3] and block structures determined by the DSORG (data set organization), RECFM (record format), and other parameters
Remarkably S/360 predates UNIX by several years, so its definitely not a novel concept. Albeit apparently IBM systems cost order of magnitude more than PDPs of the time.
Of course there are also classic LISP machines where afaik pretty much everything was a SEXPR, but I have even less knowledge about the details of those.
You should check out PowerShell. It's based on exactly that concept. Everything is an object.
`Import-Csv` and `ConvertFrom-Json` return objects that you can manipulate using LINQ or PowerShell's `*-Object` cmdlets (`Select-Object`, `Where-Object`, `Group-Object`, etc.). `Get-Process` returns objects representing your system's processes, `Get-Service` returns objects for your system's services, and so on.
Being able to chain that in your pipeline is incredibly powerful. You can send data into and out of CSVs and various other providers in a single line. It's a huge productivity boost.
The concept of everything as a file, has awesome benefits I believe, but that doesn't meant files need to be unstructured text.
There's a core problem, that with text languages (like AWK or structured regex (sed, etc) you end up both having to parse AND manipulate data. which is no fun and prone to errors.
Abstracting away all of that into codecs that all of coreutils could speak would be very cool to me.
The second issue is structured data vs unstructured text. CSVs, or other table based formats make sense sometimes, and sometimes you want to be able to query things more easily. JSON provides that. What's the synthesis of both I don't know... but maybe this is closer to heaven.
I'd like to _minimize_ the amount of parsing or text manipulation done in any program and be able to focus on extracting and inserting the data I need, like a rosetta stone that just handles everything for me while I work. I want to be able to do:
awk "{print $schoolName}" or awk "{print $4}" and it just works - json, text, CSV.