#+TITLE: NFiles

NFiles is a Common Lisp library that deals with customizable path resolution,
file persistence and loading.

Its main use case is help manage user-centric files like configuration files.

In some aspects, it can be seen as "Common Lisp 'logical pathnames' over CLOS".

** Goals

- Performance ::
  - Data is not persisted to disk if it does not need to.
  - Files are read only once (unless modified externally).
- Extensibility :: Persist any data structure the way you want it.
- Reliability :: no corruption and no data loss should occur.

** Features

- Dynamic and customizable path expansion.
- Extensible serialization and deserialization.
- Cached reads and writes ::
  When a =file= object expands to the same path as another one, a =read= or
  =write= on it won't do anything in case there was no change since last write.
- (*Experimental!*) On-the-fly PGP encryption.
- Profile support.
- On read errors, existing files are backed up.
- On write errors, no file is written to disk, the existing file is preserved.
- A =remote-file= can point to a URL, which is automatically downloaded if the
  local file is not found.

** Motivation

This package was developed after dealing with a problem when delivering Common
Lisp images: when an image is generated, path expansion may already be resolved
and thus hard-coded within the image, which makes it unfit for delivery.
Consider this:

#+begin_src lisp
> (defvar *foo-path* (uiop:xdg-config-home))
*FOO-PATH*
> *foo-path*
#P"/home/johndoe/.config/"
#+end_src

Now if I ship this image to my friend Kaboom, =*foo-path*= will expand to

#+begin_src lisp
#P"/home/johndoe/.config/"
#+end_src

on their machine instead of the expected

#+begin_src lisp
#P"/home/kaboom/.config/"
#+end_src

** Examples

A basic session:

#+begin_src lisp
(defvar *config-file* (make-instance 'nfiles:config-file :base-path #p"my-app/init.lisp"))

(nfiles:expand *config-file*)
; => #P"/home/johndoe/.config/my-app/init.lisp"

(setf (nfiles:content *config-file*) "Hello file!") ; The file is written to disk.

(nfiles:content *config-file*)
; => "Hello file!"
#+end_src

The following convenience macro ensures the file is updated when done with the
body:

#+begin_src lisp
  (nfiles:with-file-content (content *config-file*)
    (format t "Length: ~a~%" (length content))
    (setf content (serapeum:string-replace "file" content "config")))
#+end_src

The =with-paths= helper allows for let-style bindings of the expanded paths:

#+begin_src lisp
  (let ((file1 (make-instance 'nfiles:file))
        (file2 (make-instance 'nfiles:file :base-path #p"alt")))
    (nfiles:with-paths ((path1 file1)
                        (path2 file2))
      (list path1 path2)))
#+end_src

A =remote-file= works the same but needs some specialization:

#+begin_src lisp
  (defmethod nfiles:fetch ((profile nfiles:profile) (file remote-counter-file) &key)
    (dex:get (nfiles:url file)))

  ;; Optional:
  (defmethod nfiles:check ((profile nfiles:profile) (file remote-counter-file) content &key)
    (let ((path (nfiles:expand file)))
      (ironclad:byte-array-to-hex-string
       (ironclad:digest-file :sha3 path))))

  (let ((file (make-instance 'nfiles:remote-file
                             ;; The URL to download from if the file is not found on disk.
                             :url (quri:uri "https://example.org")
                             ;; Without base-path, the file won't be saved to disk.
                             :base-path #p"/tmp/index.html"
                             :checksum "794df316afac91572b899b52b54f53f04ef71f275a01c44b776013573445868c95317fc4a173a973e90addec7792ff8b637bdd80b1a6c60b03814a6544652a90")))
    ;; On access, file is automatically downloaded if needed and the checksum is verified:
    (nfiles:content file)
    ;; ...
    )
#+end_src

See the [[file:package.lisp][package]] documentation for a usage guide and more examples.

** Configuration

NFiles was designed with configurability in mind.  All configuration happens through
subclassing of the =file= and =profile= classes together with method
specialization.

All configuration methods are specialized against =profile= and =file= so that
the user can easily *compose* the behaviour:
- Profile-customization impacts all files using that profile;
- File-customization impacts the files of that specific type (or subtype)
  regardless of their profile.

Of course you can specialize against both!

The specialization methods are divided into the following:

- =resolve= :: This is where path resolution is done.  On call site, prefer the
  =expand= convenience wrapper.

- =deserialize= and =serialize= :: This is how the content is transformed
  to the file on disk.  These functions are meant to be called by the
  =read-file= and =write-file= methods.

- =read-file= and =write-file= :: This is how the file is read and written to
  disk.  These functions are responsible for calling the =deserialize= and
  =serialize= methods.

- =fetch= :: This generic function is only called for =remote-file= objects.  You
  _must_ define its methods.  It does not have any method by default so as to
  not burden NFiles with undesirable dependencies.

- =check :=: Like =fetch=, this generic function is only called for =remote-file=
  objects to test the integrity of the downloaded file.  You _must_ define its
  methods.  It does not have any method by default so as to not burden NFiles
  with undesirable dependencies.

** Conditions and restarts

Some NFiles-specific conditions are raised in case of exceptional situations to
provide for interactive and customizable behaviour:

- =external-modification= :: The file was modified externally.  See the
  =on-external-modification= slot to automate what to do in this case.

- Read error restarts can also customized, see the =on-read-error= slot to
  automate what to do in this case.

- =process-error= :: This may be raised for instance when =gpg= fails to encrypt.
  The =use-recipient= restart is provided to retry with the given recipient.

** Shadowing

NFiles 1 shadows =cl:delete=, thus you should not =:use= the package (as with
any other library anyways).

** Platform support

It's pure Common Lisp and all compilers plus all operating systems should be
supported.

Some notes:

- All compilers but SBCL depend on [[https://github.com/sionescu/iolib][IOlib]] to preserve file attributes.
- Android devices also depend on [[https://github.com/sionescu/iolib][IOlib]] to preserve file attributes,
  regardless of the compiler.
- File attributes might not be preserved on Windows.

** Roadmap

- Improve PGP support.
- Support OS-level locks (à-la Emacs / LibreOffice).
- Improve portability.
  - In particular, preservation of file attributes may not work on
    Windows.
- Compressing =write-file= and =read-file= (for instance with zstd / lz).
  - But is it such a good idea?  Users should prefer compression at
    the level of the file system.

** Change log

*** 1.1.4

- Remove =NASDF= as a dependency.

*** 1.1.3

- Ensure a =cl:pathname= is returned from =resolve=.
- Complete some missing documentation.

*** 1.1.2

- Add restarter functions for =ask=, =reload=, etc.
  Mind that =cl:delete= is now shadowed (this was necessary to preserve backward
  compatibility).
  Do not =:use= the package!

- Switch from =hu.dwim.defclass-star= to [[https://github.com/atlas-engineer/nclasses/][nclasses]].

*** 1.1.1

- Allow path expansion for =virtual-file= (as in 1.0.0).

  This restores the usefulness of virtual-files, namely to handle the
  path-expansion business while deferring the read/write business to a
  third-party.

  =virtual-profile= still nullifies path expansions (as in 1.1.0).

*** 1.1.0

- Add support for Android.
- Nullify path expansion for =virtual-file= and =virtual-profile=.
- Ensure that the =deserialize= method of =virtual-file= and =virtual-profile=
  return nil.
- Fix =basename= corner case.
- Add report messages to all restarts.

** History

NFiles was originally developed for user file management in [[https://nyxt.atlas.engineer][Nyxt]], so the "N"
may stand for it, or "New", or whatever poetic meaning you may find behind it!
