bfd, ld: add CTF section linking

This is quite complicated because the CTF section's contents depend on
the final contents of the symtab and strtab, because it has two sections
whose contents are shuffled to be in 1:1 correspondence with the symtab,
and an internal strtab that gets deduplicated against the ELF strtab
(with offsets adjusted to point into the ELF strtab instead).  It is
also compressed if large enough, so its size depends on its contents!

So we cannot construct it as early as most sections: we cannot even
*begin* construction until after the symtab and strtab are finalized.
Thankfully there is already one section treated similarly: compressed
debugging sections: the only differences are that compressed debugging
sections have extra handling to deal with their changing name if
compressed (CTF sections are always called ".ctf" for now, though we
have reserved ".ctf.*" against future use), and that compressed
debugging sections have previously-uncompressed content which has to be
stashed away for later compression, while CTF sections have no content
at all until we generate it (very late).

BFD also cannot do the link itself: libctf knows how to do it, and BFD
cannot call libctf directly because libctf already depends on bfd for
file I/O.  So we have to use a pair of callbacks, one, examine_strtab,
which allows a caller to examine the symtab and strtab after
finalization (called from elf_link_swap_symbols_out(), right before the
symtabs are written, and after the strtab has been finalized), and one
which actually does the emission (called emit_ctf simply because it is
grouped with a bunch of section-specific late-emission function calls at
the bottom of bfd_elf_final_link, and a section-specific name seems best
for that).  emit_ctf is actually called *twice*: once from lang_process
if the emulation suggests that this bfd target does not examine the
symtab or strtab, and once via a bfd callback if it does.  (This means
that non-ELF targets still get CTF emitted, even though the late CTF
emission stage is never called for them).

v2: merged with non-ELF support patch: slight commit message
    adjustments.
v3: do not spend time merging CTF, or crash, if the CTF section is
    explicitly discarded.  Do not try to merge or compress CTF unless
    linking.
v4: add CTF_COMPRESSION_THRESHOLD.  Annul the freed input ctf_file_t's
    after writeout: set SEC_IN_MEMORY on the output contents so a future
    bfd enhancement knows it could free it.  Add SEC_LINKER_CREATED |
    SEC_KEEP to avoid having to add .ctf to the linker script.  Drop
    now-unnecessary ldlang.h-level elf-bfd.h include and hackery around
    it.  Adapt to elf32.em->elf.em and elf-generic.em->ldelf*.c
    changes.
v5: fix tabdamage.  Drop #inclusions in .h files: include in .c files,
    .em files, and use struct forwards instead.  Use bfd_section_is_ctf
    inline function rather than SECTION_IS_CTF macro.  Move a few
    comments.

	* Makefile.def (dependencies): all-ld depends on all-libctf.
	* Makefile.in: Regenerated.

include/
	* bfdlink.h (elf_strtab_hash): New forward.
	(elf_sym_strtab): Likewise.
	(struct bfd_link_callbacks <examine_strtab>): New.
	(struct bfd_link_callbacks <emit_ctf>): Likewise.

bfd/
	* elf-bfd.h (bfd_section_is_ctf): New inline function.
	* elf.c (special_sections_c): Add ".ctf".
	(assign_file_positions_for_non_load_sections): Note that
	compressed debugging sections etc are not assigned here.  Treat
	CTF sections like SEC_ELF_COMPRESS sections when is_linker_output:
	sh_offset -1.
	(assign_file_positions_except_relocs): Likewise.
	(find_section_in_list): Note that debugging and CTF sections, as
	well as reloc sections, are assigned later.
	(_bfd_elf_assign_file_positions_for_non_load): CTF sections get
	their size and contents updated.
	(_bfd_elf_set_section_contents): Skip CTF sections: unlike
	compressed sections, they have no uncompressed content to copy at
	this stage.
	* elflink.c (elf_link_swap_symbols_out): Call the examine_strtab
	callback right before the strtab is written out.
	(bfd_elf_final_link): Don't cache the section contents of CTF
	sections: they are not populated yet.  Call the emit_ctf callback
	right at the end, after all the symbols and strings are flushed
	out.

ld/
	* ldlang.h: (struct lang_input_statement_struct): Add the_ctf.
	(struct elf_sym_strtab): Add forward.
	(struct elf_strtab_hash): Likewise.
	(ldlang_ctf_apply_strsym): Declare.
	(ldlang_write_ctf_late): Likewise.
	* ldemul.h (ldemul_emit_ctf_early): New.
	(ldemul_examine_strtab_for_ctf): Likewise.
	(ld_emulation_xfer_type) <emit_ctf_early>: Likewise.
	(ld_emulation_xfer_type) <examine_strtab_for_ctf>: Likewise.
	* ldemul.c (ldemul_emit_ctf_early): New.
	(ldemul_examine_strtab_for_ctf): Likewise.
	* ldlang.c: Include ctf-api.h.
	(CTF_COMPRESSION_THRESHOLD): New.
	(ctf_output): New. Initialized in...
	(ldlang_open_ctf): ... this new function.  Open all the CTF
	sections in the input files: mark them non-loaded and empty
	so as not to copy their contents to the output, but linker-created
	so the section gets created in the target.
	(ldlang_merge_ctf): New, merge types via ctf_link_add_ctf and
	ctf_link.
	(ldlang_ctf_apply_strsym): New, an examine_strtab callback: wrap
	ldemul_examine_strtab_for_ctf.
	(lang_write_ctf): New, write out the CTF section.
	(ldlang_write_ctf_late): New, late call via bfd's emit_ctf hook.
	(lang_process): Call ldlang_open_ctf, ldlang_merge_ctf, and
	lang_write_ctf.
	* ldmain.c (link_callbacks): Add ldlang_ctf_apply_strsym,
	ldlang_write_ctf_late.
	* emultempl/aix.em: Add ctf-api.h.
	* emultempl/armcoff.em: Likewise.
	* emultempl/beos.em: Likewise.
	* emultempl/elf.em: Likewise.
	* emultempl/generic.em: Likewise.
	* emultempl/linux.em: Likewise.
	* emultempl/msp430.em: Likewise.
	* emultempl/pe.em: Likewise.
	* emultempl/pep.em: Likewise.
	* emultempl/ticoff.em: Likewise.
	* emultempl/vanilla.em: Likewise.
	* ldcref.c: Likewise.
	* ldctor.c: Likewise.
	* ldelf.c: Likewise.
	* ldelfgen.c: Likewise.
	* ldemul.c: Likewise.
	* ldexp.c: Likewise.
	* ldfile.c: Likewise.
	* ldgram.c: Likewise.
	* ldlex.l: Likewise.
	* ldmain.c: Likewise.
	* ldmisc.c: Likewise.
	* ldver.c: Likewise.
	* ldwrite.c: Likewise.
	* lexsup.c: Likewise.
	* mri.c: Likewise.
	* pe-dll.c: Likewise.
	* plugin.c: Likewise.

	* ldelfgen.c (ldelf_emit_ctf_early): New.
	(ldelf_examine_strtab_for_ctf): tell libctf about the symtab and
	strtab.
	(struct ctf_strsym_iter_cb_arg): New, state to do so.
	(ldelf_ctf_strtab_iter_cb): New: tell libctf about
	each string in the strtab in turn.
	(ldelf_ctf_symbols_iter_cb): New, tell libctf
	about each symbol in the symtab in turn.
	* ldelfgen.h (struct elf_sym_strtab): Add forward.
	(struct elf_strtab_hash): Likewise.
	(struct ctf_file): Likewise.
	(ldelf_emit_ctf_early): Declare.
	(ldelf_examine_strtab_for_ctf): Likewise.
	* emultempl/elf-generic.em (LDEMUL_EMIT_CTF_EARLY): Set it.
	(LDEMUL_EXAMINE_STRTAB_FOR_CTF): Likewise.
	* emultempl/aix.em (ld_${EMULATION_NAME}_emulation): Add
	emit_ctf_early and examine_strtab_for_ctf, NULL by default.
	* emultempl/armcoff.em (ld_${EMULATION_NAME}_emulation): Likewise.
	* emultempl/beos.em (ld_${EMULATION_NAME}_emulation): Likewise.
	* emultempl/elf.em (ld_${EMULATION_NAME}_emulation): Likewise.
	* emultempl/generic.em (ld_${EMULATION_NAME}_emulation): Likewise.
	* emultempl/linux.em (ld_${EMULATION_NAME}_emulation): Likewise.
	* emultempl/msp430.em (ld_${EMULATION_NAME}_emulation): Likewise.
	* emultempl/pe.em (ld_${EMULATION_NAME}_emulation): Likewise.
	* emultempl/pep.em (ld_${EMULATION_NAME}_emulation): Likewise.
	* emultempl/ticoff.em (ld_${EMULATION_NAME}_emulation): Likewise.
	* emultempl/vanilla.em (ld_vanilla_emulation): Likewise.

	* Makefile.am: Pull in libctf (and zlib, a transitive requirement
	for compressed CTF section emission).  Pass it on to DejaGNU.
	* configure.ac: Add AM_ZLIB.
	* aclocal.m4: Added zlib.m4.
	* Makefile.in: Regenerated.
	* testsuite/ld-bootstrap/bootstrap.exp: Use it when relinking ld.

ld/ldlang.c

@@ -26,6 +26,7 @@
 #include "safe-ctype.h"
 #include "obstack.h"
 #include "bfdlink.h"
+#include "ctf-api.h"
 
 #include "ld.h"
 #include "ldmain.h"
@@ -128,6 +129,7 @@ bfd_boolean delete_output_file_on_failure = FALSE;
 struct lang_phdr *lang_phdr_list;
 struct lang_nocrossrefs *nocrossref_list;
 struct asneeded_minfo **asneeded_list_tail;
+static ctf_file_t *ctf_output;
 
  /* Functions that traverse the linker script and might evaluate
     DEFINED() need to increment this at the start of the traversal.  */
@@ -151,6 +153,12 @@ int lang_statement_iteration = 0;
 
 #define SECTION_NAME_MAP_LENGTH (16)
 
+/* CTF sections smaller than this are not compressed: compression of
+   dictionaries this small doesn't gain much, and this lets consumers mmap the
+   sections directly out of the ELF file and use them with no decompression
+   overhead if they want to.  */
+#define CTF_COMPRESSION_THRESHOLD 4096
+
 void *
 stat_alloc (size_t size)
 {
@@ -3597,6 +3605,186 @@ open_input_bfds (lang_statement_union_type *s, enum open_bfd_mode mode)
     einfo ("%F");
 }
 
+/* Open the CTF sections in the input files with libctf: if any were opened,
+   create a fake input file that we'll write the merged CTF data to later
+   on.  */
+
+static void
+ldlang_open_ctf (void)
+{
+  int any_ctf = 0;
+  int err;
+
+  LANG_FOR_EACH_INPUT_STATEMENT (file)
+    {
+      asection *sect;
+
+      /* Incoming files from the compiler have a single ctf_file_t in them
+	 (which is presented to us by the libctf API in a ctf_archive_t
+	 wrapper): files derived from a previous relocatable link have a CTF
+	 archive containing possibly many CTF files.  */
+
+      if ((file->the_ctf = ctf_bfdopen (file->the_bfd, &err)) == NULL)
+	{
+	  if (err != ECTF_NOCTFDATA)
+	    einfo (_("%P: warning: CTF section in `%pI' not loaded: "
+		     "its types will be discarded: `%s'\n"), file,
+		     ctf_errmsg (err));
+	  continue;
+	}
+
+      /* Prevent the contents of this section from being written, while
+	 requiring the section itself to be duplicated in the output.  */
+      /* This section must exist if ctf_bfdopen() succeeded.  */
+      sect = bfd_get_section_by_name (file->the_bfd, ".ctf");
+      sect->size = 0;
+      sect->flags |= SEC_NEVER_LOAD | SEC_HAS_CONTENTS | SEC_LINKER_CREATED;
+
+      any_ctf = 1;
+    }
+
+  if (!any_ctf)
+    {
+      ctf_output = NULL;
+      return;
+    }
+
+  if ((ctf_output = ctf_create (&err)) != NULL)
+    return;
+
+  einfo (_("%P: warning: CTF output not created: `s'\n"),
+	 ctf_errmsg (err));
+
+  LANG_FOR_EACH_INPUT_STATEMENT (errfile)
+    ctf_close (errfile->the_ctf);
+}
+
+/* Merge together CTF sections.  After this, only the symtab-dependent
+   function and data object sections need adjustment.  */
+
+static void
+lang_merge_ctf (void)
+{
+  asection *output_sect;
+
+  if (!ctf_output)
+    return;
+
+  output_sect = bfd_get_section_by_name (link_info.output_bfd, ".ctf");
+
+  /* If the section was discarded, don't waste time merging.  */
+  if (output_sect == NULL)
+    {
+      ctf_file_close (ctf_output);
+      ctf_output = NULL;
+
+      LANG_FOR_EACH_INPUT_STATEMENT (file)
+	{
+	  ctf_close (file->the_ctf);
+	  file->the_ctf = NULL;
+	}
+      return;
+    }
+
+  LANG_FOR_EACH_INPUT_STATEMENT (file)
+    {
+      if (!file->the_ctf)
+	continue;
+
+      /* Takes ownership of file->u.the_ctfa.  */
+      if (ctf_link_add_ctf (ctf_output, file->the_ctf, file->filename) < 0)
+	{
+	  einfo (_("%F%P: cannot link with CTF in %pB: %s\n"), file->the_bfd,
+		 ctf_errmsg (ctf_errno (ctf_output)));
+	  ctf_close (file->the_ctf);
+	  file->the_ctf = NULL;
+	  continue;
+	}
+    }
+
+  if (ctf_link (ctf_output, CTF_LINK_SHARE_UNCONFLICTED) < 0)
+    {
+      einfo (_("%F%P: CTF linking failed; output will have no CTF section: %s\n"),
+	     ctf_errmsg (ctf_errno (ctf_output)));
+      if (output_sect)
+	{
+	  output_sect->size = 0;
+	  output_sect->flags |= SEC_EXCLUDE;
+	}
+    }
+}
+
+/* Let the emulation examine the symbol table and strtab to help it optimize the
+   CTF, if supported.  */
+
+void
+ldlang_ctf_apply_strsym (struct elf_sym_strtab *syms, bfd_size_type symcount,
+			 struct elf_strtab_hash *symstrtab)
+{
+  ldemul_examine_strtab_for_ctf (ctf_output, syms, symcount, symstrtab);
+}
+
+/* Write out the CTF section.  Called early, if the emulation isn't going to
+   need to dedup against the strtab and symtab, then possibly called from the
+   target linker code if the dedup has happened.  */
+static void
+lang_write_ctf (int late)
+{
+  size_t output_size;
+  asection *output_sect;
+
+  if (!ctf_output)
+    return;
+
+  if (late)
+    {
+      /* Emit CTF late if this emulation says it can do so.  */
+      if (ldemul_emit_ctf_early ())
+	return;
+    }
+  else
+    {
+      if (!ldemul_emit_ctf_early ())
+	return;
+    }
+
+  /* Emit CTF.  */
+
+  output_sect = bfd_get_section_by_name (link_info.output_bfd, ".ctf");
+  if (output_sect)
+    {
+      output_sect->contents = ctf_link_write (ctf_output, &output_size,
+					      CTF_COMPRESSION_THRESHOLD);
+      output_sect->size = output_size;
+      output_sect->flags |= SEC_IN_MEMORY | SEC_KEEP;
+
+      if (!output_sect->contents)
+	{
+	  einfo (_("%F%P: CTF section emission failed; output will have no "
+		   "CTF section: %s\n"), ctf_errmsg (ctf_errno (ctf_output)));
+	  output_sect->size = 0;
+	  output_sect->flags |= SEC_EXCLUDE;
+	}
+    }
+
+  /* This also closes every CTF input file used in the link.  */
+  ctf_file_close (ctf_output);
+  ctf_output = NULL;
+
+  LANG_FOR_EACH_INPUT_STATEMENT (file)
+    file->the_ctf = NULL;
+}
+
+/* Write out the CTF section late, if the emulation needs that.  */
+
+void
+ldlang_write_ctf_late (void)
+{
+  /* Trigger a "late call", if the emulation needs one.  */
+
+  lang_write_ctf (1);
+}
+
 /* Add the supplied name to the symbol table as an undefined reference.
    This is a two step process as the symbol table doesn't even exist at
    the time the ld command line is processed.  First we put the name
@@ -7585,6 +7773,8 @@ lang_process (void)
   if (config.map_file != NULL)
     lang_print_asneeded ();
 
+  ldlang_open_ctf ();
+
   bfd_section_already_linked_table_free ();
 
   /* Make sure that we're not mixing architectures.  We call this
@@ -7661,6 +7851,14 @@ lang_process (void)
 	}
     }
 
+  /* Merge together CTF sections.  After this, only the symtab-dependent
+     function and data object sections need adjustment.  */
+  lang_merge_ctf ();
+
+  /* Emit the CTF, iff the emulation doesn't need to do late emission after
+     examining things laid out late, like the strtab.  */
+  lang_write_ctf (0);
+
   /* Copy forward lma regions for output sections in same lma region.  */
   lang_propagate_lma_regions ();