#+TITLE: Nyxt Developer's Manual

# Please install org-make-toc so the the TOC below will be automatically
# generated.
# https://github.com/alphapapa/org-make-toc
* Table of contents                                                     :TOC:
:PROPERTIES:
:TOC:      :include all :ignore this
:END:
:CONTENTS:
- [[#development-environment-guide][Development environment guide]]
- [[#developers-installation-guide][Developer's installation guide]]
  - [[#installing-sbcl][Installing SBCL]]
  - [[#lisp-dependencies][Lisp dependencies]]
  - [[#gtk-dependencies][GTK dependencies]]
    - [[#gnulinux-freebsd-gtk-dependencies][GNU/Linux, FreeBSD GTK dependencies]]
  - [[#fetch-the-source-code][Fetch the source code]]
  - [[#compile][Compile]]
    - [[#using-the-makefile][Using the Makefile]]
    - [[#using-lisp][Using Lisp]]
  - [[#interacting-with-a-compiled-version-of-nyxt][Interacting with a compiled version of Nyxt]]
    - [[#slime][SLIME]]
    - [[#sly][SLY]]
- [[#contributing][Contributing]]
  - [[#asking-for-help][Asking for help]]
    - [[#issue-tracker][Issue tracker]]
    - [[#learning-common-lisp][Learning Common Lisp]]
    - [[#irc-channel][IRC Channel]]
  - [[#commit-style][Commit style]]
  - [[#branch-management][Branch management]]
  - [[#adding-a-new-lisp-dependency][Adding a new Lisp dependency]]
  - [[#programming-conventions][Programming conventions]]
  - [[#irc-channel][IRC Channel]]
:END:
* Development environment guide

If you are new to developing Nyxt, you may want a straightforward way to
set up the environment so that you can hack on Nyxt with minimal friction.
This is one way to achieve that goal. Keep in mind that this part of the
manual is highly opinionated in it's choice of tools. If you wish to use
other tools, refer to the rest of the manual below.

To proceed, make sure that you have the following installed:
- [[https://www.sbcl.org/][SBCL]]
- [[https://guix.gnu.org][The GUIX package manager]]
- [[https://www.gnu.org/software/emacs/][Emacs]]
- [[https://github.com/joaotavora/sly][SLY]]

Now that you have those working on your system, clone the Nyxt repository into
=~/common-lisp=. Then, in your Emacs configuration file, add the following and
restart Emacs:

#+begin_src lisp
(load "~/common-lisp/nyxt/build-scripts/nyxt-guix.el" :noerror)

(setq sly-lisp-implementations
        '((nyxt-sbcl
           (lambda () (nyxt-make-guix-cl-for-nyxt
                  "~/common-lisp/nyxt"
                  :force t
                  :cl-implementation "sbcl"
                  :cl-system "nyxt/gi-gtk"
                  :no-grafts t
                  :ad-hoc '("emacs" "xdg-utils" "git"))))))
#+end_src

Next, in Emacs, run ~M-- M-x sly RET nyxt-sbcl RET~ to start the SLY REPL.
Wait for it to finish and the REPL will open. At this point you are almost ready
to start hacking. In the SLY REPL, write the following:

#+begin_src lisp
(asdf:load-system :nyxt/gi-gtk)
(nyxt:start)
#+end_src

With Nyxt started in this way, you can hack on Nyxt without recompiling the
whole codebase or even restarting the browser. You can make your changes,
recompile only the relevant file/expression with SLY and it's done.

You can also run the tests locally with:

#+begin_src lisp
(asdf:test-system :nyxt)
; and
(asdf:test-system :nyxt/gi-gtk)
#+end_src

Keep in mind that it is recommended to restart the SLY session before and
after running the tests.

* Developer's installation guide

There are several ways to install Nyxt on your system. For example, to install
with Guix, follow the instructions in the [[../build-scripts/nyxt.scm]] file of the
Nyxt repository. In this section, we will provide a more tool-agnostic way.

Nyxt is written in Common Lisp.  It should build with any standard Common Lisp
implementation but currently, only [[http://www.sbcl.org/][SBCL]] support is tested. It is designed to be
cross-platform, cross-engine compatible.  Nyxt is available in both WebKit and
WebEngine (experimental) flavors.

** Installing SBCL

You'll need SBCL ≥ 2.0.0 to compile Nyxt.

You can obtain SBCL from your package manager or by downloading it directly from
the SBCL repository.

To install SBCL from source, download SBCL:
[[http://www.sbcl.org/platform-table.html]].  Full installation instructions can be
found here: [[http://www.sbcl.org/getting.html]].

** Lisp dependencies

Nyxt requires some Lisp libraries and, since some are pinned at specific
versions, relying on Quicklisp is discouraged.

All Lisp dependencies are included as a submodule of this repository, so unless
the submodules are disabled (i.e. ~NYXT_SUBMODULES=false~), there is nothing
more to be done.

Otherwise clone the required repositories listed in =.gitmodules= into
=~/common-lisp= (or [[https://www.common-lisp.net/project/asdf/asdf.html#Configuring-ASDF-to-find-your-systems][another directory where ASDF will find it]]).  For instance,
to get the latest version of =cl-webkit=, you would typically run the following:

#+begin_src sh
mkdir -p ~/common-lisp
git clone https://github.com/joachifm/cl-webkit ~/common-lisp/cl-webkit
#+end_src

** GTK dependencies

*** GNU/Linux, FreeBSD GTK dependencies

- WebKitGTK+ also known as webkit2gtk (make sure to use the most recent version
  for security reasons)
- gobject-introspection (for WebKitGTK+ bindings)
- glib-networking (for WebKitGTK+)
- gsettings-desktop-schemas (for WebKitGTK+)
- libfixposix
- xclip (for clipboard support)
- enchant (for spellchecking)
- pkg-config (for web-extensions)

- Debian-based distributions:
  #+begin_src sh
  sudo apt install sbcl libwebkit2gtk-4.0-dev gobject-introspection glib-networking gsettings-desktop-schemas libfixposix-dev pkg-config xclip enchant-2 libssl-dev
  #+end_src

- Arch Linux:
  #+begin_src sh
  sudo pacman -S git sbcl cl-asdf webkit2gtk gobject-introspection glib-networking gsettings-desktop-schemas enchant libfixposix
  #+end_src

- Fedora:
  #+begin_src sh
  sudo dnf install sbcl webkit2gtk3-devel glib-networking gsettings-desktop-schemas libfixposix-devel xclip enchant pkgconf
  #+end_src

- FreeBSD and derivatives
  #+begin_src sh
  pkg install sbcl webkit2-gtk3 glib-networking libfixposix xclip enchant rubygem-pkg-config
  #+end_src

If your distribution does not install libraries in an [[https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard][FHS]]-expected location, you
have to let your Lisp compiler know where to find them.  To do so, add the
library directories to ~cffi:*foreign-library-directories*~ list.  For instance,
if you are running Guix you may want to expose =~/.guix-profile/lib= to the
compiler by adding the following snippet to =~/.sbclrc=:

#+begin_src lisp
(require "asdf")

(let ((guix-profile (format nil "~a/.guix-profile/lib/" (uiop:getenv "HOME"))))
  (when (and (probe-file guix-profile)
             (ignore-errors (asdf:load-system "cffi")))
    (push guix-profile
          (symbol-value (find-symbol (string '*foreign-library-directories*)
                                     (find-package 'cffi))))))
#+end_src

#+begin_quote
A note of caution about installing WebKit via your package manager: Your
distribution supplied version of WebKit may not provide up-to-date versions of
WebKit including the latest security patches.  WebKitGTK+ tries to do the best
job possible with maintaining security patches upstream, but it is also up to
the the distribution provider to update their packages to take advantage of
these fixes.
#+end_quote

** Fetch the source code

Clone the Nyxt repository into =~/common-lisp= (or [[https://www.common-lisp.net/project/asdf/asdf.html#Configuring-ASDF-to-find-your-systems][another directory where ASDF
will find it]]):

#+begin_src sh
mkdir -p ~/common-lisp
git clone --recurse-submodules https://github.com/atlas-engineer/nyxt ~/common-lisp/nyxt
#+end_src

** Compile
*** Using the Makefile

The following command will build the Lisp core.

- GNU/Linux:
  #+begin_src sh
  make all
  #+end_src

- FreeBSD
  #+begin_src sh
  gmake all
  #+end_src

Inside the Makefile you'll find many options you can specify. Run ~make~
to display some documentation or see the [[../makefile][Makefile]] for more details.

*** Using Lisp

Start your Lisp and run the following commands:

#+NAME: compile
#+begin_src lisp
(asdf:make :nyxt/gtk-application)
#+end_src

Your Lisp implementation must have produced an executable in the directory where
the =.asd= file is located.

** Interacting with a compiled version of Nyxt

After compiling Nyxt or installing it in some other way, you can use SLIME or
SLY to interact with it in a REPL. This is accomplished by starting a ~swank~
or ~slynk~ server (for SLIME and SLY respectively) from Nyxt and connecting
to it through Emacs.

*** SLIME

1. Run the command =start-swank= in Nyxt. Note the port number in the message
   buffer. The default is 4006.
2. Connect to the ~swank~ server in Emacs with
   ~M-x slime-connect RET 127.0.0.1 RET 4006~.

*** SLY

1. Run the command =start-slynk= in Nyxt. Note the port number in the message
   buffer. The default is 4006.
2. Connect to the ~slynk~ server in Emacs with
   ~M-x sly-connect RET 127.0.0.1 RET 4006~.

* Contributing

Nyxt is a joint effort and we need you to make it succeed!  You can find ideas
[[https://github.com/atlas-engineer/nyxt/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-issue][on our issue tracker]] to suit your interests and skills. When ready to start
working please fork the repository, add your changes and open a pull request on
GitHub to pass the review process.  Refer to the [[*Branch management][branch management section]] for
more detailed information.

You can contribute to Nyxt without commit access.  However, if you're a frequent
contributor, you may request it.  Remember that with great power comes great
responsibility.

** Asking for help

Feel free to contact us at any point if you need guidance. There are several
ways to ask for help from the community.

*** Issue tracker

The first and easiest one is to simply [[https://github.com/atlas-engineer/nyxt/issues][open up an issue]] with whatever problem or
suggestion you wish to discuss.

*** Learning Common Lisp

See https://nyxt.atlas.engineer/learn-lisp for a few recommendations.

*** IRC Channel

You can find Nyxt on Libera IRC: [[https://kiwiirc.com/nextclient/irc.libera.chat/nyxt][#nyxt]].

** Commit style

We follow the general Git guidelines, namely we try to commit atomic changes
that are "clean", that is, on which Nyxt builds and starts.

Make sure to make seperate commits in these cases to avoid distracting noise in
commits with actual changes:

- Indentation and whitespace trimming;

- Code movements (within a file or to a different file).  In this case, it's
  crucial that the commit contains nothing else, otherwise "diffs" may fail to
  highlight the changes.

For commit messages, we follow (somewhat flexibly) the convention of prefixing
the title with the basename of the file that was modified.  For instance, for
changes in =source/mode/blocker.lisp= the commit message would look like this:

#+begin_example
mode/blocker: What and why this change.

Rest of the message here.
#+end_example

Your commit should clarify _what_ it does and _why_ (in case it's not already
obvious).

** Branch management

Nyxt uses the following branches:

- =master= for development;
- =<feature-branches>= for working on particular features;
- =<2,3,...>-series= to backport commits corresponding to specific major
  versions.

It's recommended to branch off from the target branch and to rebase onto it
right before merging.  This keeps the history as clear as possible and reduces
the complexity of the diff.

Unless the changes are trivial and each commit is atomic (that is, leaving Nyxt
fully functional), they should be followed by a merge commit.  That is
guaranteed by using the merge option =no-ff= (no fast-forward).  If required,
the merge commit can be reworded.

The names of the branches really matter since the merge commit references them,
so please take that into account!

After the changes are merged, please do not forget to delete obsolete or
dangling branches. If you merge the remote branch instead of the local one, then
GitHub deletes the remote branch automatically.

Note to core contributors: since you have commit access, you can push trivial
changes directly to the target branch (skipping the review process).  The merge
commit is required when at least one commit isn't atomic.

** Adding a new Lisp dependency
- Add and shallow clone upstream source as a Git submodule in [[../_build/]]
  directory.
- Add dependency name to [[../nyxt.asd]] and [[SOURCES.org][documents/SOURCES.org]].
- Add dependency to [[../build-scripts/nyxt.scm]], [[https://guix.gnu.org/en/packages/][checking]] to make sure Guix
  already has it packaged.

** Programming conventions

We try to follow the usual Common Lisp conventions as recommended by
[[https://www.cs.umd.edu/~nau/cmsc421/norvig-lisp-style.pdf][Norvig & Pitman's Tutorial on Good Lisp Programming Style]] and [[https://google.github.io/styleguide/lispguide.xml][Google Common Lisp
Style Guide]].

For symbol naming conventions, see https://www.cliki.net/Naming+conventions.

We've also developed some of our own:

- Prefer =first= and =rest= over =car= and =cdr= respectively.
- Use =define-class= instead of =defclass=.
- Use =nyxt:define-package= for Nyxt-related pacakges.  Notice that it features
  default imports (e.g. =export-always=) and package nicknames (e.g. =alex=,
  =sera=, etc.).  Prefer =uiop:define-package= for general purpose packages.
- Export using =export-always= (from Serapeum) next to the symbol definition.
  This helps prevent exports to go out-of-sync, or catch typos.
  Unlike =export=, =export-always= saves you from surprises upon recompilation.
- When sensible, declaim the function types using =->= (from Serapeum).  Note
  that there is then no need to mention the type of the arguments and the return
  value in the docstring.
- Use the =maybe= and =maybe*= types instead of =(or null ...)= and =(or null
  (array * (0)) ...)= respectively.
- Use the =list-of= type for typed lists.
- We make heavy use of Alexandria and Serapeum, remember to use them instead of
  writing the same boilerplate over and over.  In particular, note these
  systematic uses of Serapeum:
  - =sera:eval-always=;
  - =export-always=;
  - =sera:and-let*=;
  - =sera:lret=;
  - =sera:single=
  - =->= (declaimed types).
- Use =funcall*= to not error when function does not exist.
- Prefer classes over structs.  Rationale:
  - Class slots have documentation.
  - Class allow for full-fledged CLOS use (metaclasses, etc.).
  - Structs have read-only slots but it's easy enough to implement them for
    classes.
  - Structs have better performance, but this is usually micro-optimization, and
    even then class implementations can be made more efficient via MOP.
- Classes should be usable with just a =make-instance=.
- Slots classes should be formatted in the following way:
#+begin_src lisp
(slot-name
 slot-value
 ...
 :documentation "Foo.")
#+end_src

When =slot-value= is the only parameter specified then:
#+begin_src lisp
(slot-name slot-value)
#+end_src
- Prefer =defmethod= over =defun= if one of the arguments is a user-class.  This
  allows the user to write specializations of subclasses.
- =customize-instance= is reserved for end users.  Use
  =initialize-instance :after= or =slot-unbound= to initialize the slots.
  Set up the rest of the class in =customize-instance :after=.  Bear in mind
  that anything in this last method won't be customizable for the end user.
- Almost all files should be handled via the =nfiles= library.
- Specialize =print-object= for recurring class instances.
- =(setf SLOT-WRITER) :after= is reserved for "watchers",
  i.e. handlers that are run whenever the slot is set.  The =:around= method is
  not used by watchers, and thus the watcher may be overridden.
- A function as a slot value is often a sign that it should be a method instead.
  Methods give more flexibility to the end user.
  Example: Avoid adding a =constructor= slot, make it a method instead.
- Define generic functions (in particular if they are heavily used) using an
  explicit call to =defgeneric=, not with just calls to =defmethod=.  This
  enables proper source location of the generic function (otherwise it cannot be
  found), plus it lets you write different documentation for the generic and the
  specialized methods.
- We use the =%foo%= naming convention for special local variables. But special
  variables are rare and ideally they should be avoided.
- We suffix predicates with =-p=.  Unlike the usual convention, we always use a
  hyphen even for single word predicates.
- Prefer the term =url= over =uri=.
- URLs should be of type =quri:uri=.  If you need to manipulate a URL string, call
  it =url-string=. In case the value contains a URL, but is not =quri:url=, use
  =url-designator= and its =url= method to normalize into =quri:uri=.
- Paths should be of type =cl:pathname=.
  Use =uiop:native-namestring= to "send" to OS-facing functions,
  =uiop:ensure-pathname= to "receive" from OS-facing functions or to "trunamize".
- Prefer =handler-bind= over =handler-case=: when running from the REPL, this
  triggers the debugger with a full stacktrace; when running the Nyxt binary,
  all conditions are caught anyway.
- Do not handle the =T= condition, this may break everything.  Handle =error=,
  =serious-condition=, or exceptionally =condition= (for instance if you do not
  control the called code, and some libraries subclass =condition= instead of
  =error=).
- Dummy variables are called =_=.
- Prefer American spelling.

# - Conversion functions =FROM->TO= or =->TO= for generic functions.  The
#   only one that comes to mind is =url= which does not follow this convention...

# - Blocking function should be prefixed with =wait-on-=.

** IRC Channel

#+html: You can find Nyxt on Libera IRC: <a href="https://kiwiirc.com/nextclient/irc.libera.chat/nyxt">#nyxt</a>

# Local Variables:
# eval: (add-hook 'before-save-hook
#                 (lambda nil (if (fboundp 'org-make-toc)
#                                 (org-make-toc)
#                                 (message-box "Please install org-make-toc.")))
#                 nil
#                 t)
# End:
