{"id":11999,"date":"2022-06-19T19:34:39","date_gmt":"2022-06-20T02:34:39","guid":{"rendered":"https:\/\/www.oeconomist.com\/blogs\/daniel\/?p=11999"},"modified":"2022-07-08T15:57:38","modified_gmt":"2022-07-08T22:57:38","slug":"virtual-shelving","status":"publish","type":"post","link":"https:\/\/www.oeconomist.com\/blogs\/daniel\/?p=11999","title":{"rendered":"Virtual Shelving"},"content":{"rendered":"

[This entry was revised and expanded on 2022:07\/07.]<\/p>

I am always uncomfortable with the process of organizing books and articles on shelves or in boxes. I desire to have them grouped by each author and by each subject of interest; these desires cannot be reconciled without having multiple copies of each book and of each article, which multiplicity I cannot afford.<\/p>

Electronic copies are a different matter. Even without multiple copies, symbolic links, which I discussed in a previous entry<\/a>, make it possible effectively to list the same file in multiple directories. Hereïn, I'll explain the principle structure that I use for organizing documents, and I'll present some small utilities that facilitate creating and maintaining that structure on POSIX<\/abbr>-compliant file systems. This structure is not as fine-grained as might be imagined, but it strikes a balance appropriate to my purposes. (For a more sophisticated system one should employ an application storing and retrieving documents mediated by a cataloguing relational database.)<\/p>

As with many systems, mine have each a directory named Documents<\/code><\/q>. Its two subdirectories relevant to this discussion are Authors<\/code> and Subjects<\/code>. \"\"<\/p>

The entries in Subjects<\/code> are subdirectories with names such as Economics<\/code><\/q>, Logic and Probability<\/code><\/q>, Mathematics<\/code><\/q>, and Philosophy<\/code><\/q>. \"\"<\/p>

In turn, the entries in each of these are subdirectories with the names of authors. \"\"<\/p>

Finally, in each of these subdirectories are entries for files containing their work corresponding to the superdirectory<\/em>. For example, Documents\/Subjects\/Logic and Probability\/Johnson William Ernest\/<\/code> would have entries for works by him on logic or on probability, \"\" but his article on indifference curves would be listed instead in Documents\/Subjects\/Economics\/Johnson William Ernest\/<\/code>.<\/p>

Most of the subdirectories of Authors<\/code> have names<\/em> corresponding to the subdirectories in the third level of the Subjects<\/code> substructure, but all of these subdirectories in Authors<\/code> are different directories from those in the Subjects<\/code> substructure. \"\"<\/p>

Each of most of these subdirectories of Authors<\/code> lists not subdirectories nor files, but symbolic links<\/em>. \"\" These links take their names from the subdirectories of Subjects<\/code>, but they do not link to those subdirectories. Instead, each links to an author-specific sub-subdirectory. Thus, for example, Documents\/Authors\/Johnson William Ernest\/Logic and Probability<\/code> is a symbolic link to Documents\/Subjects\/Logic and Probability\/Johnson William Ernest<\/code>. It is as if<\/em> the subject-specific collection of an author's works is the author-specific collection of works on that subject, just as it should be<\/em>. \"\"<\/p>

One could, instead, use the complementary organization, in which the Subjects<\/code> substructure were ultimately dependent upon the Authors<\/code> substructure, or use a hybrid organization in which some of the dependency flows one way and some the other. The determinant should be what is most important to preserve if the collection is copied to a file system that does not support symbolic links, as in the case of a SD<\/abbr> card with a FAT<\/abbr> file system.<\/p>

I've sketched the principal structure, but want to note useful complications of two sorts.<\/p>

The first is that symbolic links may be used to place some subjects effectively under others. For example, logic an probability fall within the scope of philosophy. As well as having a directory named Logic and Probability<\/code><\/q> listed in Subjects<\/code>, I have a symbolic link to it listed in Philosophy<\/code>. \"\" Indeed, when a subject falls within the intersection of other subjects, each may have such a symbolic link, and I have links to Documents\/Subjects\/Logic and Probability<\/code> not only in Philosophy<\/code> but in Mathematics<\/code> and in Economics<\/code>.<\/p>

The second is that symbolic links may be used effectively to list a document with multiple authors in the directory for each author. \"\" And essentially the same device may be used to classify a single document under different subjects.<\/p>

Although this organization is not especially fine-grained, it requires the creation of many<\/em> directories and symbolic links. I've written seven utilities in Python<\/a> to reduce the burden. Two of those utilities were presented in a previous 'blog entry<\/a> because they can be put to more general purpose. Here, I will present five more.<\/p>

(Again, these utilities are written for POSIX<\/abbr>-compliant file systems. Windows is not POSIX<\/abbr>-compliant.<\/em> A full discussion of the relevant issues would be tedious, as would be an effort to rewrite these programs to support Windows.)<\/p>

<\/strong>The first of these programs, mkdocdir.py<\/code>, takes two or three arguments, and creates corresponding directories and symbolic links. The first argument is the name of an author, and the second is the name of a subject. A command mkdocdir.py \"De Morgan Augustus\" Mathematics<\/code><\/span> will create directories Documents<\/code>, Documents\/Authors<\/code>, Documents\/Subjects<\/code>, Documents\/Authors\/De Morgan Augustus<\/code>, Documents\/Subjects\/Mathematics<\/code>, and Documents\/Subjects\/Mathematics\/De Morgan<\/code> unless they already exist, and will create the symbolic link Documents\/Authors\/De Morgan Augustus\/Mathematics<\/code> (to Documents\/Subjects\/Mathematics\/De Morgan<\/code>) unless it already exists. The optional third argument, uses an alternative to Documents<\/code>, but is otherwise the same; a command mkdocdir.py \"De Morgan Augustus\" Mathematics MyDocs<\/code><\/span> will create MyDocs\/Subjects\/Mathematics\/De Morgan Augustus<\/code> &c.<\/p> 

#!\/usr\/bin\/env python\r\nimport os\r\nimport sys\r\n\r\nsubjectSource = True\r\n\r\nargc = len(sys.argv)\r\nif argc < 3 or argc > 4:\r\n    prog_name = sys.argv[0][sys.argv[0].rindex(\"\/\")+1:]\r\n    print(\"Syntax: \" + prog_name\r\n                     + \" <author-name> <subject> [<documents-directory>]\\n\"\r\n                     +\r\n          \"<documents-directory> defaults to \\\"Documents\\\".\\n\"\r\n                     +\r\n          \"Examples: \" + prog_name\r\n                       + \" \\\"De Morgan Augustus\\\" Mathematics\\n\"\r\n                     +\r\n          \"          \" + prog_name\r\n                       + \" Menger_Carl Economics MyDocs\")\r\nelse:\r\n    if argc < 4:\r\n        dir_documents = os.path.expanduser(\"~\") + \"\/Documents\"\r\n    else:\r\n        dir_documents = sys.argv[3]\r\n    os.makedirs(dir_documents,0o777,True)\r\n    os.chdir(dir_documents)\r\n    if subjectSource == True:\r\n        dir_sources = \"Subjects\"\r\n        dir_links = \"Authors\"\r\n        d3 = sys.argv[1]\r\n        d4 = sys.argv[2]\r\n    else:\r\n        dir_sources = \"Authors\"\r\n        dir_links = \"Subjects\"\r\n        d3 = sys.argv[2]\r\n        d4 = sys.argv[1]\r\n    source = dir_sources + \"\/\" + d4 + \"\/\" + d3\r\n    link = dir_links + \"\/\" + d3 + \"\/\" + d4\r\n    os.makedirs(source,0o777,True)\r\n    os.makedirs(dir_links + \"\/\" + d3,0o777,True)\r\n    if not os.path.exists(link):\r\n        os.chdir(dir_links + \"\/\" + d3)\r\n        os.symlink(\"..\/..\/\" + dir_sources + \"\/\" + d4 + \"\/\" + d3,d4)<\/pre><\/div> 
The second, cpdoc.py<\/code>, copies a file into the structure, creating directories and symbolic links as necessary; it thus combines the functionality of creating the relevant directories and symbolic link with that of copying a file into the structure.  cpdoc.py<\/code> takes three or four arguments.  The first is a file-path, the second the name of an author, and the third the name of a subject.  The optional fourth again specifies an alternative to Documents<\/code>.  A command cpdoc.py \"A Treatise on Probability.pdf\" \"Keynes John Maynard\" \"Logic and Probability\"<\/code><\/span> will create the relevant directories and symbolic link, and then copy a file named A Treatise on Probability.pdf<\/code><\/q> from the present working directory into Documents\/Subjects\/Logic and Probability\/Keynes John Maynard<\/code>.<\/p> 
#!\/usr\/bin\/env python\r\nimport os\r\nimport sys\r\nimport shutil\r\n\r\nsubjectSource = True\r\n\r\nargc = len(sys.argv)\r\nif argc < 4 or argc > 5:\r\n    prog_name = sys.argv[0][sys.argv[0].rindex(\"\/\")+1:]\r\n    print(\"Syntax: \" + prog_name\r\n                     +\r\n          \" <file> <author-name> <subject> [<documents-directory>]\\n\"\r\n                     +\r\n          \"<documents-directory> defaults to \\\"Documents\\\".\\n\"\r\n                     +\r\n          \"Examples: \" + prog_name\r\n                       +\r\n          \" ~\/Downloads\/On_the_Syllogism.pdf \\\"De Morgan Augustus\\\" \\\"Logic\\\"\\n\"\r\n                     +\r\n          \"          \" + prog_name\r\n                       +\r\n          \" introductorylec02whatgoog.pdf Whately_Richard Economics MyDocs\")\r\nelse:\r\n    if argc < 5:\r\n        dir_documents = os.path.expanduser(\"~\") + \"\/Documents\"\r\n    else:\r\n        dir_documents = sys.argv[4]\r\n    os.makedirs(dir_documents,0o777,True)\r\n    file = os.path.abspath(sys.argv[1])\r\n    os.chdir(dir_documents)\r\n    if subjectSource == True:\r\n        dir_sources = \"Subjects\"\r\n        dir_links = \"Authors\"\r\n        d3 = sys.argv[2]\r\n        d4 = sys.argv[3]\r\n    else:\r\n        dir_sources = \"Authors\"\r\n        dir_links = \"Subjects\"\r\n        d3 = sys.argv[3]\r\n        d4 = sys.argv[2]\r\n    source = dir_sources + \"\/\" + d4 + \"\/\" + d3\r\n    link = dir_links + \"\/\" + d3 + \"\/\" + d4\r\n    os.makedirs(source,0o777,True)\r\n    os.makedirs(dir_links + \"\/\" + d3,0o777,True)\r\n    if not os.path.exists(link):\r\n        os.chdir(dir_links + \"\/\" + d3)\r\n        os.symlink(\"..\/..\/\" + dir_sources + \"\/\" + d4 + \"\/\" + d3,d4)\r\n     shutil.copy2(file,dir_documents + \"\/\" + source)<\/pre><\/div> 
The third, mvdoc.py<\/code>, is the same as cpdoc.py<\/code>, except that mvdoc.py<\/code> moves<\/em> the file specified into the structure, instead of copying it.<\/p> 
#!\/usr\/bin\/env python\r\nimport os\r\nimport sys\r\nimport shutil\r\n\r\nsubjectSource = True\r\n\r\nargc = len(sys.argv)\r\nif argc < 4 or argc > 5:\r\n    prog_name = sys.argv[0][sys.argv[0].rindex(\"\/\")+1:]\r\n    print(\"Syntax: \" + prog_name\r\n                     +\r\n          \" <file> <author-name> <subject> [<documents-directory>]\\n\"\r\n                     +\r\n          \"<documents-directory> defaults to \\\"Documents\\\".\\n\"\r\n                     +\r\n          \"Examples: \" + prog_name\r\n                       +\r\n          \" ~\/Downloads\/On_the_Syllogism.pdf \\\"De Morgan Augustus\\\" \\\"Logic\\\"\\n\"\r\n                     +\r\n          \"          \" + prog_name\r\n                       +\r\n          \" introductorylec02whatgoog.pdf Whately_Richard Economics MyDocs\")\r\nelse:\r\n    if argc < 5:\r\n        dir_documents = os.path.expanduser(\"~\") + \"\/Documents\"\r\n    else:\r\n        dir_documents = sys.argv[4]\r\n    os.makedirs(dir_documents,0o777,True)\r\n    file = os.path.abspath(sys.argv[1])\r\n    os.chdir(dir_documents)\r\n    if subjectSource == True:\r\n        dir_sources = \"Subjects\"\r\n        dir_links = \"Authors\"\r\n        d3 = sys.argv[2]\r\n        d4 = sys.argv[3]\r\n    else:\r\n        dir_sources = \"Authors\"\r\n        dir_links = \"Subjects\"\r\n        d3 = sys.argv[3]\r\n        d4 = sys.argv[2]\r\n    source = dir_sources + \"\/\" + d4 + \"\/\" + d3\r\n    link = dir_links + \"\/\" + d3 + \"\/\" + d4\r\n    os.makedirs(source,0o777,True)\r\n    os.makedirs(dir_links + \"\/\" + d3,0o777,True)\r\n    if not os.path.exists(link):\r\n        os.chdir(dir_links + \"\/\" + d3)\r\n        os.symlink(\"..\/..\/\" + dir_sources + \"\/\" + d4 + \"\/\" + d3,d4)\r\n     shutil.move(file,dir_documents + \"\/\" + source)<\/pre><\/div> 
The fourth places a symbolic links to a specific work in another directory.  It can be used effectively to list a single work under multiple directories, associated with different authors or with different subjects or with both.  lndoc.py<\/code> takes three or four arguments.  The first is a file-path, the second the name of an author, and the third the name of a subject.  The optional fourth again specifies an alternative to Documents<\/code>.  A command lndoc.py \"Reconsideration of the Theory of Value.pdf\" \"Allen Roy George Douglas\" Economics<\/code><\/span> will create the relevant directories and symbolic link to a directory, and then create a symbolic link to a file named Reconsideration of the Theory of Value.pdf<\/code><\/q> from the present working directory into Documents\/Subjects\/Economic\/Allen Roy George Douglas<\/code>.  It will also create the associated directories and a symbolic link Documents\/Authors\/Allen Roy George Douglas\/Economics<\/code> to Documents\/Subjects\/Economics\/Allen Roy George Douglas<\/code> if these do not already exist.<\/p> 
#!\/usr\/bin\/env python\r\nimport os\r\nimport sys\r\nimport shutil\r\n\r\nsubjectSource = True\r\n\r\nargc = len(sys.argv)\r\nif argc < 4 or argc > 5:\r\n    prog_name = sys.argv[0][sys.argv[0].rindex(\"\/\")+1:]\r\n    print(\"Syntax: \" + prog_name\r\n                     +\r\n          \" <file> <author-name> <subject> [<documents-directory>]\\n\"\r\n                     +\r\n          \" defaults to \\\"Documents\\\".\\n\"\r\n                     +\r\n          \"Examples: \" + prog_name\r\n                       +\r\n          \" ~\/Downloads\/On_the_Syllogism.pdf \\\"De Morgan Augustus\\\" \\\"Logic\\\"\\n\"\r\n                     +\r\n          \"          \" + prog_name\r\n                       +\r\n          \" introductorylec02whatgoog.pdf Whately_Richard Economics MyDocs\")\r\nelse:\r\n    if argc < 5:\r\n        dir_documents = os.path.expanduser(\"~\") + \"\/Documents\"\r\n    else:\r\n        dir_documents = sys.argv[4]\r\n    os.makedirs(dir_documents,0o777,True)\r\n    file = os.path.abspath(sys.argv[1])\r\n    os.chdir(dir_documents)\r\n    if subjectSource == True:\r\n        dir_sources = \"Subjects\"\r\n        dir_links = \"Authors\"\r\n        d3 = sys.argv[2]\r\n        d4 = sys.argv[3]\r\n    else:\r\n        dir_sources = \"Authors\"\r\n        dir_links = \"Subjects\"\r\n        d3 = sys.argv[3]\r\n        d4 = sys.argv[2]\r\n    source = dir_sources + \"\/\" + d4 + \"\/\" + d3\r\n    link = dir_links + \"\/\" + d3 + \"\/\" + d4\r\n    os.makedirs(source,0o777,True)\r\n    os.makedirs(dir_links + \"\/\" + d3,0o777,True)\r\n    if not os.path.exists(link):\r\n        os.chdir(dir_links + \"\/\" + d3)\r\n        os.symlink(\"..\/..\/\" + dir_sources + \"\/\" + d4 + \"\/\" + d3,d4)\r\n    os.symlink(file,dir_documents + \"\/\" + source + \"\/\" + os.path.basename(file))\r\n<\/documents-directory><\/pre><\/div> 
Each of the above four programs contains a line subjectSource = True<\/code><\/q>.  If this line is changed to subjectSource = False<\/code><\/q> then the aforemention complementary structure is created, with the Subjects<\/code> substructure dependent upon the Authors<\/code> substructure.  The code could be modified so that a command-line switch determined which organization were used.<\/p> 
The program mkdoclnk.py<\/code> is used to create the dependent substructure from the independent substructure.  That is to say that if the documents are stored in the Subjects<\/code> substructure and the Authors<\/code> substructure has not been created or not fully created, then mkdoclnk.py<\/code>.  If the documents are stored in the Authors<\/code> substructure, then mkdoclnk.py<\/code> can create a dependent Subjects<\/code> substructure.  If the substructures have different names (such as Autori<\/code><\/q> and Materia<\/code><\/q>), then mkdoclnk.py<\/code> can accept those.  If a hybrid system is to be used, two passes of mkdoclnk.py<\/code> can complete it.<\/p>  mkdoclnk.py<\/code> takes zero to three arguments.  If the first is not given, then the source substructure is assumed to be in a directory named Subjects<\/code><\/q>.  If the second is not given, then the dependent substructure is assumed to be in a directory named Authors<\/code><\/q>.  And if the third argument is not given, them the source and dependent substructures are assumed to be in a directory named Documents<\/code><\/q>.  A command mkdoclnk.py<\/code><\/span> constructs the Authors<\/code> substructure in Documents<\/code> from the Subjects<\/code> substructure in Documents<\/code>.  A command mkdoclnk.py Pundits Topics Yappings<\/code><\/span> constructs a Topics<\/code> substructure in Yappings<\/code> from a Pundits<\/code> substructure in Yappings<\/code>. 
#!\/usr\/bin\/env python\r\nimport os\r\nimport sys\r\n\r\nsys.argc = len(sys.argv)\r\nif sys.argc > 4:\r\n    prog_name = sys.argv[0][sys.argv[0].rindex(\"\/\")+1:]\r\n    print(\"Syntax:\\n \" + prog_name\r\n                     + \" [<sources-directory> [<links-directory> [<parent-directory>]]]\\n\"\r\n                     +\r\n          \"Defaults: \" + prog_name + \" Subjects Authors ~\/Documents\\n\"\r\n                     +\r\n          \"Example: \" + prog_name + \" Authors Subjects\")\r\nelse:\r\n    if sys.argc < 4:\r\n        os.chdir(os.path.expanduser(\"~\") + \"\/Documents\")\r\n    else:\r\n        os.chdir(sys.argv[3])\r\n    dir_home = os.getcwd()\r\n    if sys.argc < 3:\r\n        dir_links = \"Authors\"\r\n    else:\r\n        dir_links = sys.argv[2]\r\n    if sys.argc < 2:\r\n        dir_source = \"Subjects\"\r\n    else:\r\n        dir_source = sys.argv[1]\r\n    list_dir_source = [entry for entry in os.scandir(\".\/\" + dir_source)\r\n                 if entry.is_dir() and not os.path.islink(entry)]\r\n    for entry in list_dir_source:\r\n        list_dir_links = [sub_entry for sub_entry in os.scandir(entry.path)\r\n                     if sub_entry.is_dir() and not os.path.islink(sub_entry)]\r\n        for sub_entry in list_dir_links:\r\n            dir_candidate = \".\/\" + dir_links + \"\/\" + sub_entry.name\r\n            if not os.path.exists(dir_candidate):\r\n                os.makedirs(dir_candidate,0o777,True)\r\n            if not os.path.exists(dir_candidate + \"\/\" + entry.name):\r\n                os.chdir(dir_candidate)\r\n                os.symlink(\"..\/..\/\" + dir_source\r\n                           + \"\/\" + entry.name + \"\/\" + sub_entry.name,\r\n                           entry.name)\r\n                os.chdir(dir_home)<\/pre><\/div>","protected":false},"excerpt":{"rendered":"[This entry was revised and expanded on 2022:07\/07.] I am always uncomfortable with the process of organizing books and articles on shelves or in boxes. I desire to have them grouped by each author and by each subject of interest; these desires cannot be reconciled without having multiple copies of each book and of each […]","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[69,4],"tags":[1639,1638,1642,1643,1641,1640,250,1637,1635,1636],"class_list":["post-11999","post","type-post","status-publish","format-standard","hentry","category-information-technology","category-public","tag-document-management","tag-documents","tag-e-books","tag-ebooks","tag-electronic-books","tag-electronic-documents","tag-programming","tag-python","tag-symbolic-links","tag-symlinks"],"_links":{"self":[{"href":"https:\/\/www.oeconomist.com\/blogs\/daniel\/index.php?rest_route=\/wp\/v2\/posts\/11999","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.oeconomist.com\/blogs\/daniel\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.oeconomist.com\/blogs\/daniel\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.oeconomist.com\/blogs\/daniel\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.oeconomist.com\/blogs\/daniel\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=11999"}],"version-history":[{"count":22,"href":"https:\/\/www.oeconomist.com\/blogs\/daniel\/index.php?rest_route=\/wp\/v2\/posts\/11999\/revisions"}],"predecessor-version":[{"id":12037,"href":"https:\/\/www.oeconomist.com\/blogs\/daniel\/index.php?rest_route=\/wp\/v2\/posts\/11999\/revisions\/12037"}],"wp:attachment":[{"href":"https:\/\/www.oeconomist.com\/blogs\/daniel\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=11999"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.oeconomist.com\/blogs\/daniel\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=11999"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.oeconomist.com\/blogs\/daniel\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=11999"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}