mk-mk.in: EXT_CFLAGS: remove .eh frames!..

[s-mailx.git] / TODO

TODO reminder.

Backward-compatibility breakers
-------------------------------

- Recipients specified on the command line should be added to those
  specified in the message when the -t option is set.

- The -q option makes me sad as it doesn't use *indentprefix* for the
  quoted file.  So either there should be -Q which does so, or -q should be
  changed.  Also see ~R below.
  [Note: i think i go for the latter.  Please complain.]

- At least optionally disallow silent discarding of invalid addresses,
  i.e., cause sending to be aborted if not all recipient addresses pass the
  validity test.

- Ditto if a resource file can't be found that has been explicitly set via
  environment variables there should be some feedback.

- I.e., it is fine to be silent unless an error occurs, but then please
  report errors and offer (in interactive mode) the possibility to act at
  a glance.

-- While there.  There should be some kind of "verbose" switch that - in
   interactive mode - also gives *positive* feedback, as in "added
   attachment X, charset Y", but without giving details about protocol
   delivery etc.

   [It is terrible that there is almost no feedback in the UI.  When
   i temporarily implemented a sorted cmdtab i've often used wrong commands,
   but got no feedback at all!  E.g., wanted to "undelete 14", first did "u
   14", then "und 14" and then realized my fault and did "undelete 14".
   *Nothing*.]

- POSIX says that, when written to DEAD: "If the file exists, the message
  shall be written to replace the contents of the file".  This is mentioned
  for ASYNCHRONOUS EVENTS, but it's the only description of what should be
  done in which way to DEAD.  savedeadletter() yet appends.  See ZOMBIE ,)

- Furthermore, *all* file operations yet append, even recipient target
  files are appended.  I don't know if this is really desirable behaviour,
  but i have not thought about that for real.  Maybe this should be at
  least configurable.

- Maybe we should not use pipes to pagers at all.
  This would also (beside getting rid of longjumps that cross allocations
  afaik etc.) make it possible to honour the *crt* variable in respect
  to what really is displayed, not in respect to message size.
  It irritates me that a message with 5 visible lines but 115 header lines
  goes through the pager, even if i have *crt=*.
  P.S.: we could simply count the headers in addition?

- The IMAP cache is file-based, which is quite nice, since a filesystem
  is a database.  On the other hand it may be better if we could hook
  into sqlite3, which is available almost everywhere by default, with
  a clearly documented DB content (so that users can use sqlite3(1) to
  dig into it).  (And just in case S-nail will ever be able to read news
  and/or have MTA functionality.)
  One question would be wether compression should be applied.

- We should possibly get away of using command line utilities for
  compression.  (At least optionally?)  Instead we should link against
  zlib(3), bz2lib(3) and lzma(3), if found.  Or we may use dlopen(3)
  instead, if found, to avoid linking (though those libraries don't need
  much linker work unless actually used afaik, 'should look in detail).
  We should also drop lzw.c, it is used for the IMAP cache.

- We should maybe turn -~ into the meaning "force interactive".
  We should extend cc-test.sh, then, to test some interactive things.

Non-breakers
------------

- We need a "void" box that can be jumped to, i.e., a state in which no box
  at all is active.

  (schdir(): realpath() local files before leaving CWD.., 2013-01-08)
  did a first step to avoid "getting stuck" when the current folder becomes
  unaccessible.
  That is however only a command-specific workaround for a deeper design
  problem.

-- When a MBOX mailbox is removed while it is opened then changing the
  folder is not possible.  (At least when the only mail that was present
  therein has been "move"d off that MBOX.)  While trying this i got
  a deallocation abort trap in lex.c:commands():385:
        if (linebuf)
                free(linebuf);
  Note that Berkeley Mail (NetBSD mail(1) that is) seems to have the same
  problem in respect to not allowing a hard-change, but only allows an exit
  (not a quit thus).  The error may still exist even there, it maybe that
  it simply doesn't show up because a stack buffer is used ...

  In interactive mode, dump the temporary file to the TMPDIR and inform the
  user; it's up to him to decide what should happen.

  Otherwise it must be possible to switch the folder (to the *void* one)
  forcefully.

-- Also, when the folder is modified concurrently we should bail, or, in an
  interactive session, prompt the user what to do.

- IDNA decoding.  It may be that this will never be supported.  But
  wouldn't it be nice for at least viewing messages?

- Mail-Followup-To:.

- Make it possible to reply to/save/write/xy part X.Y[.Z] by allowing its
  specification directly, as in, e.g., ':w 1.2'.  If doing so on an
  embedded message/rfc822, e.g., a message embedded in a digest, it should
  be possible to reply to the very message in respect to its header fields,
  but (optionally?) keep the original Cc:'d.  (Parts by Martin Neitzel)

-- In general i like the possibility of mutt(1) to toggle in between header
   and header-ignore mode with a simple switch (h), and that *this toggle
   is used when replying*.  Yes i know mutt(1) sucks beside that in respect
   to base64, MBOX, QP for CRLF files etc.

-- Also, mutt(1) quotes all text parts of a message, not only the first
   one!  This should at least be optionally available.

- If pipes fail for part viewers then at least the usual PART X.Y should be
  shown, maybe even including some error message.
  I had 'set pipe-text/html="lynx -dump -force_html /dev/stdin"' but NetBSD
  does not have lynx(1), and i thought i've found a S-nail(1) bug.
  And so it was.

= Update: (Change "Part X:" display message.., 2012-12-20) did one step,
  but we need to rewrite the entire layers to handle that the separating
  newline in between messages is considered to be part of it.

- I want to have a ~R tilde command that works like ~r except it performs
  quoting of the input just as ~m does.  Also see -q above.

- Offer the possibility to work with certificate fingerprints instead of
  full certificates, in equal spirit to the current maintainers S-Postman
  and Mercurial.  S-nail(1) could simply offer something in equal spirit to
  the formers --fingerprint, so that no other tool is necessary for
  certificate management (for at least secure transport).

- It would be nice if it would be possible to define a format string for
  *quote*, like 'set quote="format=some formats"'.
  In general the current approach is somewhat messy IMO.  I.e., it would
  make more sense to act rather like mutt(1) and as written elsewhere in
  this document, i.e., have some toggles that act on the display and use it
  for multiple modes (show/reply/forward etc.)
  Otherwise introduce commands which include all the headers plus, e.g.,
  "hreply" or "freply", and then the ditto series, i.e., "hReply" ...

-- This would also mean that interactive message editing would work
  accordingly.  PleasE!

- Command line editing should gain possibility of context sensitive tab
  completion.

-- Note that the TTY is sick.  If ^C on input it simply jumps to next
   input, instead of saying "Interrupted, one more to die hard" or
   something (talking about ~@ charset selection prompts in particular).

- For those who use S-nail(1) only with a MTA it may be desirable to have
  some "smopts" expansion mechanism in equal spirit to NetBSD mailx(1).

- While talking about NetBSD mailx(1), the author can imagine that being
  able to use the optional -H:xy stuff is sometimes nice.

- Check against RFC 5322.
  Rework all the header parsing code.  Actually understand the content,
  classify the stuff so that it matches what is defined in RFC dependent on
  header field.  Place the result in objects that know what they represent.
  See the name extraction topic below.

- Also check against newest POSIX.
  Maybe add a bash(1)ish POSIXLY_CORRECT (or so) variable?  Though..

- Maybe there should be an additional ZOMBIE directive that is served in
  equal spirit to DEAD, but that could be a valid MBOX... ?
  What i want is a *real* resend, best if possible from command line.
  Meaning, also the possibility to postpone a message.  In general.

- POP3 doesn't support "newmail" for real.  If implemented, should it sync?
  Look at POP3 impl. in general..

- Having a newsreader would be a really cool thing.  (RFC 977 and 2980)

- There should be a way to ignore the From_ line, as opposed to the From:
  line, i.e., distinctively.

- There should be a variable that controls wether leading and trailing
  empty lines of parts and/or messages as such should be printed or not.

- RFC 2387: multipart/related.

- rfc2384.txt etc.  I.e., Much better URL support.

- printhead()/hprf(): support %n newline format (%t tab?).
  Make it possible to use the *datefield* algorithm for plain From_ derived
  dates (needs a From_ parser, i.e., strptime()-alike).
  Once we have that, rename *datefield-markout-older* to
  *date-markout-older* ??
  Note that NetBSD's mail(1) has some other nice things.
  Note also that our code is quite unflexible.

-- NetBSD's mail(1) has nice *indentprefix* and *indentpostscript*
   variables (though prefix and appendix or prefix and suffix, but..).
   Note that our code is quite unflexible.

- The "top" command should honour ignoretab, or there should be a very
  special "top" ignoretab.  It simply doesn't make sense to "top" 5 lines
  when all that you get are Received: lines...

- In the very end it is not that hard to add (optional) MTA
  functionality at a most simple level.
  Use sqlite for aliases (and possibly cache), then.

- We should support IMAP compression over the wire.

Low-Level
---------

- Revise the code:
  + [v13:messed up] Use NetBSD KNF in respect to parens for return+.
  + [v13:started] Add C99-likeish typedefs and use them everywhere.
  + [v13:started] Don't use magic constants/values.
  + [v13:started] Use const arguments whenever possible.  Yet started, but
    with ugly casts at some places because this is a can of worms.
  + Inline functions?  [Restrict pointers?]
  + [v13:started] Document the functions in the interface declarators.
  + [v13:started] Resort the functions and where they are implemented.
  + Make more use of struct str, and offer more support functions for it.
  + Make more use of value carriers in the call stack.  There are functions
    with an incredible amount of arguments, needlessly.
  ++ E.g., use a struct readlinectx.
  + Ftemp(): offer an operation mode enum that includes "do-unlink".
    In general: now there is O_CLOEXEC etc. ...
  + Lazy install sigchild handler etc.

- [v13:started] Improve name extraction rules.  And field parsing.  There
  are structured and unstructured fields.  There are quoted pairs and
  comments etc.  Rewrite the entire parsing mechanism to comply to RFC
  5322, and try to merge all those many subparsers around in the codebase,
  and accordingly.  So much duplicated work ...
  Name parsing has been improved a bit for v13, but it's still broken.
  yankword(), *extract(), etc.: RFC 5322 says that comments in address
  fields SHOULD NOT be used (mutt(1) maps them to full name-addr forms if
  approbiate, even if that actually changes content!!?), and that full
  name-addr SHOULD be used.  Our functions are yet quite silly (i.e.,
  leading comments remain, as in "(bier2) <a2@b2.de>", unless the address
  doesn't come in angle brackets, trailing go away, as in "<a6@b6.de>
  (bier6)", that becomes "<a6@b6.de>").
  Something silly like
    (co$mm1) abc@däf.de (cö,mm,2)   ('c'o"m"m.3)
  Should eventually become
    co$mm1 cö,mm,2 'c'o"m"m.3 <abc@xn--df-via.de>
  on the display, or, with IDNA decoding (and thus rather unlikely)
    co$mm1 cö,mm,2 'c'o"m"m.3 <abc@däf.de>
  It should NOT become this mutt(1)ism:
    "co$mm1 cö,mm,2 'c'omm.3" <abc@däf.de>
  Or?

- Think about a name bypass hashmap cache, and whenever we have to skin or
  nalloc() or whatever, look in there.  Maybe even an additional link for
  non GFULL(/GSKIN) and fully skinned struct name objects.
  The amount of duplicated work in this codebase is frustrating, but the
  real healing would make necessary a complete rewrite of the name handling!
  Such a cache would work without touching the current code flow ... or
  allow a smooth transition to a new one anyway.

- Think about a value() cache.  (Yet added boption() and soption() stubs.)

- Think about replacing the IMAP cache with an SQLITE3 interface.
  Or rewrite it.  Error handling etc. etc. is peculiar.

- The char classification stuff can be improved; currently each character
  has exactly one classification bit set, even if multiple would apply
  (e.g., HT=\t == CNTRL|SPACE|ASCII|BLANK).  This would allow better
  testing using our own classification functions in quite some places.

- The quoted-printable Content-Transfer-Encoding: supports soft linebreaks;
  it happens that a lot of mailers (Apple Mail?, Microsoft Word, Yahoo!
  Webmail) create HTML parts which solely consist of a single line,
  created via soft linebreaks.
  To handle such mess we need to be able to break out of the input-line ==
  output line relationship that is still fixated in the codebase.
  I.e., it is not even sufficient to convert "rest" into an array, but best
  would be if we would be able to sequentially work what we have, and
  detect when it is safe to "dump that out".
  This MUST be part of the send/mime layer rewrite in 15.0.

- I want a clean PTF interface for the actual layers.  There should be no
  switch() statements around that test for the type of BOX that is
  currently open.  Especially important for possible NEWS support, but the
  code is a mess in general...

- I hate longjmp()s and signals.  I suspect some pitfalls in the codebase.
  All that stuff should at least be collected in a single place.
  Just imagine how easy it could be done with a non-blocking select(2)
  based event loop, ISIG off termios flags, etc.  ach!!

  I would like to see that compilation with a C++ compiler is possible,
  though that would be a long way and be especially problematic due to the
  (C ish) way enums are used.

- I never used anything but the *datefield* option, and it would really be
  nice if the date strings would be parsed off into some 16 byte or what
  storage when about to producing the summary, so that it would be directly
  available and there would be no need to reread the mail.  Moreover, or
  even more than that - the m_date field exists and should possibly simply
  be init, at least in these cases.  (P.S.: this doesn't contradict the
  statement somewhere else in this file that the structure should be
  slacked; simply use multiple thereof or so)

- All error messages should not go to stderr but instead we should add our
  own n_warn() family and use that.  In the background we should have
  a ring of error messages (oldest fall off), and a command that is capable
  to display the ring.  The command loop should recognize whenever an error
  happened during the last command, and print something like "XY errors
  occurred", followed by a (truncated as necessary) error report.
  It simply doesn't make any sense to print errors on stderr if normal
  output goes to stdout and scrolls it off the screen.
  Note that yet some errors messages still go to stdout.

- Dramatically pimp the cc-test.sh script, and recognize and report
  compilation errors, too.
  WE NEED MORE AUTOMATIZED TESTING!

- An `account' should imply an environment snapshot of all variables.
  I.e., variables which become modified while an account is active would
  gain their old / a different value (back) when switching off this
  account.  We would need some "set-variable-global" or something, to
  get around that?
  There should be an implicit account ("global" or "*" or whatever) that
  can be switched back to.

-- It would be very nice if it would be possible to have
   account-specific macros that will be lost when a user switches off
   an account.

--- It would also be very nice if we could define macros while defining
    account, i.e., recursive definition of macros...
    See below on account/define though.

- At some later time extend the logic behind -# -- it should not have
  a current folder, but start in VOID mode (...), and unless one is
  explicitly chosen..  We need a reliable batch mode.

- (Support for mailcap files?  As of RFC 1524?  Unlikely.  Though the
  % expansions would be very helpful, especially once we can address
  individuals parts as of v15.0!)

TODO S-nail 14.5:
-----------------

- Perform a WS check, and fix it.
  (For 15.0: reformat, add vim(1) modelines and use s-it-mode.
  To do so, write a small script and ensure no functional change occurred.)

- Deal with faulty message selection that may occur when selecting threads
  via & (when at least mixed with other selectors).

-- Also (?same problem?) the thread sort doesn't get

    [A is deleted]
    B answers A
      C answers B
      D answers B
    E is unrelated
    F answers A

  The current sort fails to recognize that F and the thread starting at
  B are related, which results in a mess.

- Drop the restriction that environment variables cannot be modified
  from within interactive S-nail(1), check carefully what depends on
  that (findmail() looks for MAIL, be aware of -u interaction etc.).
  Drop tempdir, myname and homedir variables and always use functions to
  look the values up (just make them function local variables where
  used).
  If necessary, add more hooks in vars.c:_check_special_vars() (or
  finally implement the value cache for real).
  Drop main.c:_setup_vars(), then.

- If we do link against OpenSSL, try to compile a test program and use
  the OpenSSL version of MD5 if possible.

  I think this is the right time to change our config/make system so that
  the Makefile is generated with only those files which are actually
  needed.  Move the Makefile vars to user.conf.

  (Possibly drop usage of compressions tools and link against the
  compression libraries.  Maybe dlopen() them?  Otherwise leave that for
  future version.)

- Make *password* and *password-USER@HOST* global, introduce
  *PROTOCOL-auth-password* and *PROTOCOL-auth-password-USER@HOST*.
  Then, drop smtp_auth_var(), and change lookup_password_for_token() to
  work like (PROTOCOL, user,host|TOKEN) -- *password* is a global override,
  only if that is not set lookup protocol specific.
  Use that single function from everywhere.

- Change **use-starttls* to the opposite.  I.e., CAPA or whatever, if TLS
  is supported, use automatically.  *UNLESS* explicitly disabled.

- Add a global *keepalive*, add *keepalive-USER@HOST*.  Add and use
  a generic, single function to get the value for either protocol.

- NOTE: we do not really support IPv6 sofar in that we are not prepared to
  deal with IPv6 addresses.
  Introduce an URI abstraction structure in 14.5, and start using it, i.e.,
  _pop3_user() and, very important, sopen().
  Note this shit software use a lot of places which mess around...

- The account command involves a lot of code but it's only difference to
  define is that a "fi" is executed.  If we would have the possibility to
  explicitly jump to VOID boxes, we could use QUIT->FOLDER instead of
  FOLDER[OK? -> QUIT], and then it'd be up to the use to simply use a "fi"
  at the end of a define.  Or so.
  See above for more on account/define though.

-- If *folder* is set to an IMAP box, and we're about to "mbox" data to
   there, and we're currently on a POP3 server, and the connection fails,
   we're completely lost and cannot even interrupt...

- mutt(1) dotlock ..., "mbox" command doesn'T work?

- ARGH!  Should `folders' auto-login if *folder* is an IMAP account that is
  not active?  Why does _expand() use *mailname* to expand `@', not
  getfold() (care: res may point into cbuf, savestr() or so!).
  Why does demail() etc. treat *mailname* as a file (more or less), why do
  we need *mailname* at all;  we should have Folder objects, multiple of
  which concurrently, one the active; a Folder may not become *folder*
  unless it has write (store) capabilities).  Maybe then `mbox' works fine
  if connected to a POP3 server with a *MBOX* on an IMAP account that yet
  never was connected and needs to read a password on the terminal before
  the login works ... note the latter situation yet kills us since i think
  INT is blocked during all that ;-((

- I had a connection collapse during a POP3 download, and neither was
  there a chance to get access to the 22 yet downloaded mails (after
  five minutes of waiting followed by CNTRL-C), nor did the layer
  recognize this very well (got myriads of `POP3 connection already
  closed.' messages, btw., the thirty-something messages which were not
  yet downloaded caused (after CNTRL-C) this:

    POP3 connection already closed.
    POP3 connection already closed.
    POP3 connection already closed.
    POP3 connection already closed.
    POP3 connection already closed.
    POP3 connection already closed.
     U 23                    Thu Jan  1 0    /    0
    POP3 connection already closed.
    POP3 connection already closed.
    POP3 connection already closed.
    POP3 connection already closed.
    POP3 connection already closed.
    POP3 connection already closed.
     U 24                    Thu Jan  1 0    /    0

  i had to switch off and back).  At least we didn't crash.
  Thereafter, still very unstable connection, i tried to login again:

    ? fi xpop
    Resolving host pop.gmail.com . . . done.
    Connecting to 173.194.70.109:pop3s . . . connected.
    Comparing DNS name: "pop.gmail.com"
    POP3 connection already closed.
    +OK Gpop ready for requests from XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  [..nothing!..]
    ^CInterrupt
    ? fi %
    "/private/var/mail/steffen": 0 messages
    ? fi xpop
    Resolving host pop.gmail.com . . . done.
    Connecting to 173.194.70.109:pop3s . . . connected.
    Comparing DNS name: "pop.gmail.com"
    ? fi
    ? fi
    ? fi

  Can of worms.

- Add a value-duplication command, i.e.,
    clone _x header
    unset header
    ...
    clone header _x
    unset _x

- Some (configurable?) verbosity for certificate validation.
  (Martin Neitzel)
  Rework certificate handling to match RFC 6125!  14.4!

- #include cleanup.  Much too much!

- Remove all occurrences of mbtowc() with mbrtowc(); temporarily add (some)
  global mbstate_t objects until the send / MIME layer rewrite is done and
  has the carrier.  Use flip states and add aux funs with only update the
  state+toggle on success -- CURRENTLY MBTOWC FAILURES ARE PRACTICALLY NOT
  HANDLED!!

- Ensure that `.' and EOF on a line works with all TTY modes (*ignoreeof*
  relationship, too)!  EOF conditions in general!

-- NCL / current expand-on-tab: fexpand() should take additional size_t* to
   store the number of the results OR should "return char** array", so that
   individual results can be addressed.
   Then we could simply print "\nALL-RESULTS\n" and NOT expand the current
   line if the result is ambiguous, i.e., we have more than one possible
   expansion.
   However, we would then need something to print the results page-wise,
   in case we have so many of them that they don't fit on the screen.
   ...  Etc. ...

- Add TODO notes for those RFCs:
  RFC 5322 - The basic format of email messages. 
  MIME (Multimeda) email extensions
  RFC 2405 - The format of MIME message bodies. 
  RFC 2406 - Common multimedia types. 
  RFC 2017 - URL External-Body Access-Type 
  RFC 3676 - Updates to the text/plain MIME type and extensions for flowed
    text (format=flowed). 
  RFC 2407 - Encoding of non-ASCII text in message headers. 
  RFC 2183 - The Content-Disposition Header 
  RFC 2231 - Encoding of character set and language information in MIME
    parameter values. 
  SMTP (Simple Mail Transfer Protocol)
  RFC 5321 - Simple Mail Transfer Protocol. 
  RFC 6409 - Message Submission for Mail 
  RFC 4954 - SMTP Authentication 
  RFC 3207 - SMTP over TLS 
  RFC 6152 - SMTP Service Extension for 8-bit MIME Transport 
  Post Office Protocol (POP)
  RFC 1939 - Post Office Protocol v3 
  RFC 2449 - POP3 Extensions (including SASL) 
  RFC 2595 - TLS for POP3 (among others) 
  Security
  RFC 4422 - Simple Authentication and Security layer (SASL) 
  RFC 5246 - Transport Layer Security (TLS)
  RFC 977 - Network News Transfer Protocol
  RFC 2980 - Common NNTP Extensions

- IMAP: if i `d' a message then this is a local change (in the meanwhile);
  yet, when i `u' it, that will go through the server, needlessly.

- I got an email in base64 that obviously used CRNL line endings, and once
  i've replied the CR where quoted as *control* characters.
  Get rid of those (kwcrtest.mbox; may be hard to do everywhere for some
  time). (2013-08-06)

- Yet another one:
 
  :>>> T1026 NOOP
  Received SIGPIPE during IMAP operation
  IMAP write error: error:00000000:lib(0):func(0):reason(0)

  [DOING STUFF]

  ? newmail
  >>> T1027 FETCH 51:52 (FLAGS UID)
  IMAP write error: Bad file descriptor
  >>> T1028 FETCH 51:52 (RFC822.HEADER)
  IMAP write error: Bad file descriptor
  IMAP connection closed.
  IMAP connection closed.
  IMAP connection closed.
  IMAP connection closed.
  IMAP connection closed.
  IMAP connection closed.
  IMAP connection closed.
  IMAP connection closed.
  New mail has arrived.
  Loaded 2 new messages.
  >>> T1029 FETCH 51:52 (RFC822.HEADER)
  IMAP write error: Bad file descriptor
  IMAP connection closed.
  IMAP connection closed.
  IMAP connection closed.
  IMAP connection closed.
  IMAP connection closed.
   U 51                    Thu Jan  1 0    /    0 
   IMAP connection closed.
   IMAP connection closed.
   IMAP connection closed.
   IMAP connection closed.
   IMAP connection closed.

- The codebase has a lot of problems with type mixing; int,off_t,size_t are
  often used INTERCHANGEABLY to store the same size / value; especially
  IMAP seems to be quite weird.  I've yet changed some off_t to size_t,
  which is not a good idea for 32-bit and large files, but (a) it's
  questionable wether we could handle those anyway and (b) all bets are off
  since pieces of code also use int storage for them, iirc.
  This will be a long-term problem.

- edit.c doesn't do NEED_BODY (yeah, IMAP won't work anyway).

Release S-nail v20 on 2018-03-25, the 40th anniversary of Mail.
With a clean, conforming and efficient codebase, then.

vim:set fenc=utf-8:s-ts-mode