You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

3804 lines

  1. pmacct [IP traffic accounting : BGP : BMP : IGP : Streaming Telemetry]
  2. pmacct is Copyright (C) 2003-2017 by Paolo Lucente
  3. The keys used are:
  4. !: fixed/modified feature, -: deleted feature, +: new feature
  5. 1.7.0 -- 21-10-2017
  6. + ZeroMQ integration: by defining plugin_pipe_zmq to 'true', ZeroMQ is
  7. used for queueing between the Core Process and plugins. This is in
  8. alternative to the home-grown circular queue implementation (ie.
  9. plugin_pipe_size). plugin_pipe_zmq_profile can be set to one value
  10. of { micro, small, medium, large, xlarge } and allows to select
  11. among a few standard buffering profiles without having to fiddle
  12. with plugin_buffer_size. How to compile, install and operate ZeroMQ
  13. is documented in the "Internal buffering and queueing" section of
  14. the QUICKSTART document.
  15. + nDPI integration: enables packet classification, replacing existing
  16. L7-layer project integration, and is available for pmacctd and
  17. uacctd. The feature, once nDPI is compiled in, is simply enabled by
  18. specifying 'class' as part of the aggregation method. How to compile
  19. install and operate nDPI is documented in the "Quickstart guide to
  20. packet classification" section of the QUICKSTART document.
  21. + nfacctd: introduced nfacctd_templates_file so that NetFlow v9/IPFIX
  22. templates can be cached to disk to limit the amount of lost packets
  23. due to unknown templates when nfacctd (re)starts. The implementation
  24. is courtesy by Codethink Ltd.
  25. + nfacctd: introduced support for PEN on IPFIX option templates. This
  26. is in addition to already supported PEN for data templates. Thanks
  27. to Gilad Zamoshinski ( @zamog ) for his support.
  28. + sfacctd: introduced new aggregation primitives (tunnel_src_host,
  29. tunnel_dst_host, tunnel_proto, tunnel_tos) to support inner L3
  30. layers. Thanks to Kaname Nishizuka ( @__kaname__ ) for his support.
  31. + nfacctd, sfacctd: pcap_savefile and pcap_savefile_wait were ported
  32. from pmacctd. They allow to process NetFlow/IPFIX and sFlow data
  33. from previously captured packets; these also ease some debugging by
  34. not having to resort anymore to tcpreplay for most cases.
  35. + pmacctd, sfacctd: nfacctd_time_new feature has been ported so, when
  36. historical accounting is enabled, to allow to choose among capture
  37. time and time of receipt at the collector for time-binning.
  38. + nfacctd: added support for NetFlow v9/IPFIX field types #130/#131,
  39. respectively the IPv4/IPv6 address of the element exporter.
  40. + nfacctd: introduced nfacctd_disable_opt_scope_check: mainly a work
  41. around to implementations not encoding NetFlow v9/IPIFX option scope
  42. correctly, this knob allows to disable option scope checking. Thanks
  43. to Gilad Zamoshinski ( @zamog ) for his support.
  44. + pre_tag_map: added 'source_id' key for tagging on NetFlow v9/IPFIX
  45. source_id field. Added also 'fwdstatus' for tagging on NetFlow v9/
  46. IPFIX information element #89: this implementation is courtesy by
  47. Emil Palm ( @mrevilme ).
  48. + tee plugin: tagging is now possible on NetFlow v5-v8 engine_type/
  49. engine_id, NetFlow v9/IPFIX source_id and sFlow AgentId.
  50. + tee plugin: added support for 'src_port' in tee_receivers map. When
  51. in non-transparent replication mode, use the specified UDP port to
  52. send data to receiver(s). This is in addition to tee_source_ip,
  53. which allows to set a configured IP address as source.
  54. + networks_no_mask_if_zero: a new knob so that IP prefixes with zero
  55. mask - that is, unknown ones or those hitting a default route - are
  56. not masked. The feature applies to *_net aggregation primitives and
  57. makes sure individual IP addresses belonging to unknown IP prefixes
  58. are not zeroed out.
  59. + networks_file: hooked up networks_file_no_lpm feature to peer and
  60. origin ASNs and (BGP) next-hop fields.
  61. + pmacctd: added support for calling pcap_set_protocol() if supported
  62. by libpcap. Patch is courtesy by Lennert Buytenhek ( @buytenh ).
  63. + pmbgpd, pmbmpd, pmtelemetryd: added a few CL options to ease output
  64. of BGP, BMP and Streaming Telemetry data, for example: -o supplies
  65. a b[gm]p_daemon_msglog_file, -O supplies a b[gm]p_dump_file and -i
  66. supplies b[gm]p_dump_refresh_time.
  67. + kafka plugin: in the examples section, added a Kafka consumer script
  68. using the performing confluent-kafka-python module.
  69. ! fix, BGP daemon: segfault with add-path enabled peers as per issue
  70. #128. Patch is courtesy by Markus Weber ( @FvDxxx ).
  71. ! fix, print plugin: do not update link to latest file if cause of
  72. purging is a safe action (ie. cache space is finished. Thanks to
  73. Camilo Cardona ( @jccardonar ) for reporting the issue. Also, for
  74. the same reason, do not execute triggers (ie. print_trigger_exec).
  75. ! fix, nfacctd: improved IP protocol check in NF_evaluate_flow_type()
  76. A missing length check was causing, under certain conditions, some
  77. flows to be marked as IPv6. Many thanks to Yann Belin for his
  78. support resolving the issue.
  79. ! fix, print and SQL plugins: optimized the cases when the dynamic
  80. filename/table has to be re-evaluated. This results in purge speed
  81. gains when the dynamic part is time-related and nfacctd_time_new is
  82. set to true.
  83. ! fix, bgp_daemon_md5_file: if the server socket is AF_INET and the
  84. compared peer address in MD5 file is AF_INET6 (v4-mapped v6), pass
  85. it through ipv4_mapped_to_ipv4(). Also if the server socket is
  86. AF_INET6 and the compared peer addess in MD5 file is AF_INET, pass
  87. it through ipv4_to_ipv4_mapped(). Thanks to Paul Mabey for reporting
  88. the issue.
  89. ! fix, nfacctd: improved length checks in resolve_vlen_template() to
  90. prevent SEGVs. Thanks to Josh Suhr and Levi Mason for their support.
  91. ! fix, nfacctd: flow stitching, improved flow end time checks. Thanks
  92. to Fabio Bindi ( @FabioLiv ) for his support resolving the issue.
  93. ! fix, amqp_common.c: amqp_persistent_msg now declares the RabbitMQ
  94. exchange as durable in addition to marking messages as persistent;
  95. this is related to issue #148.
  96. ! fix, nfacctd: added flowset count check to existing length checks
  97. for NetFlow v9/IPFIX datagrams. This is to avoid logs flooding in
  98. case of padding. Thanks to Steffen Plotner for reporting the issue.
  99. ! fix, BGP daemon: when dumping BGP data at regular time intervals,
  100. dump_close message contained wrongly formatted timestamp. Thanks to
  101. Yuri Lachin for reporting the issue.
  102. ! fix, MySQL plugin: if --enable-ipv6 and sql_num_hosts set to true,
  103. use INET6_ATON for both v4 and v6 addresses. Thanks to Guy Lowe
  104. ( @gunkaaa ) for reporting the issue and his support resolving it.
  105. ! fix, 'flows' primitive: it has been wired to sFlow so to count Flow
  106. Samples received. This is to support Q21 in FAQS document.
  107. ! fix, BGP daemon: Extended Communities value was printed with %d
  108. (signed) format string instead of %u (unsigned), causing issue on
  109. large values.
  110. ! fix, aggregate_primitives: improved support of 'u_int' semantics for
  111. 8 bytes integers. This is in addition to already supported 1, 2 and
  112. 4 bytes integers.
  113. ! fix, pidfile: pidfile created by plugin processes was not removed.
  114. Thanks to Yuri Lachin for reporting the issue.
  115. ! fix, print plugin: checking non-null file descriptor before setvbuf
  116. in order to prevent SEGV. Similar checks were added to prevent nulls
  117. be input to libavro calls when Apache Avro output is selected.
  118. ! fix, SQL plugins: MPLS aggregation primitives were not correctly
  119. activated in case sql_optimize_clauses was set to false.
  120. ! fix, building system: reviewed minimum requirement for libraries,
  121. removed unused m4 macros, split features in plugins (ie. MySQL) and
  122. supports (ie. JSON).
  123. ! fix, sql_history: it now correctly honors periods expressed is 's'
  124. seconds.
  125. ! fix, BGP daemon: rewritten bgp_peer_print() to be thread safe.
  126. ! fix, pretag.h: addressed compiler warning on 32-bit architectures,
  127. integer constant is too large for "long" type. Thanks to Stephen
  128. Clark ( @sclark46 ) for reporting the issue.
  129. - MongoDB plugin: it is being discontinued since the old Mongo API is
  130. not supported anymore and there has never been enough push from the
  131. community to transition to the new/current API (which would require
  132. a rewrite of most of the plugin). In this phase-1 the existing
  133. MongoDB plugin is still available using 'plugins: mongodb_legacy'
  134. in the configuration.
  135. - Packet classification basing on the L7-filter project is being
  136. discontinued (ie. 'classifiers' directive). This is being replaced
  137. by an implementation basing on the nDPI project. As part of this
  138. also the sql_aggressive_classification knob has been discontinued.
  139. - tee_receiver was part of the original implementation of the tee
  140. plugin, allowing to forward to a single target and hence requiring
  141. multiple plugins instantiated, one per target. Since 0.14.3 this
  142. directive was effectively outdated by tee_receivers.
  143. - tmp_net_own_field: the knob has been discontinued and was allowing
  144. to revert to backward compatible behaviour of IP prefixes (ie.
  145. src_net) being written in the same field as IP addresses (ie.
  146. src_host).
  147. - tmp_comms_same_field: the knob has been discontinued and was
  148. allowing to revert to backward compatible behaviour of BGP
  149. communities (standard and extended) being writeen all in the same
  150. field.
  151. - plugin_pipe_amqp and plugin_pipe_kafka features were meant as an
  152. alternative to the homegrown queue solution for internal messaging,
  153. ie. passing data from the Core Process to Plugins, and are being
  154. discontinued. They are being replaced by a new implementation,
  155. plugin_pipe_zmq, basing on ZeroMQ.
  156. - plugin_pipe_backlog was allowing to keep an artificial backlog of
  157. data in the Core Process so to maximise bypass poll() syscalls in
  158. plugins. If home-grown queueing is found limiting, instead of
  159. falling back to such strategies, ZeroMQ queueing should be used.
  160. - pmacctd: deprecated support for legacy link layers: FDDI, Token Ring
  161. and HDLC.
  162. 1.6.2 -- 21-04-2017
  163. + BGP, BMP daemons: introduced support for BGP Large Communities IETF
  164. draft (draft-ietf-idr-large-community). Large Communities are stored
  165. in a variable-length field. Thanks to Job Snijders ( @job ) for his
  166. support.
  167. + BGP daemon: implemented draft-ietf-idr-shutdown. The draft defines a
  168. mechanism to transmit a short freeform UTF-8 message as part of a
  169. Cease NOTIFICATION message to inform the peer why the BGP session is
  170. being shutdown or reset. Thanks to Job Snijders ( @job ) for his
  171. support.
  172. + tee plugin, pre_tag_map: introduced support for inspetion of specific
  173. flow primitives and selective replication over them. The primitives
  174. supported are: input and output interfaces, source and destination
  175. MAC addresses, VLAN ID. The feature is now limited to sFlow v5 only.
  176. Thanks to Nick Hilliard and Barry O'Donovan for their support.
  177. + Added src_host_pocode and dst_host_pocode primitives, pocode being a
  178. compact and (de-)aggregatable (easy to identify districts, cities,
  179. metro areas, etc.) geographical representation, based on the Maxmind
  180. v2 City Database. Thanks to Jerred Horsman for his support.
  181. + Kafka support: introduced support for user-defined (librdkafka) config
  182. file via the new *_kafka_config_file config directives. Full pathname
  183. to a file containing directives to configure librdkafka is expected.
  184. All knobs whose values are string, integer, boolean are supported.
  185. + AMQP, Kafka plugins: introduced new directives kafka_avro_schema_topic,
  186. amqp_avro_schema_routing_key to transmit Apache Avro schemas at regular
  187. time intervals. The routing key/topic can overlap with the one used to
  188. send actual data.
  189. + AMQP, Kafka plugins: introduced support for start/stop markers when
  190. encoding is set to Avro (ie. 'kafka_output: avro'); also Avro schema
  191. is now embedded in a JSON envelope when sending it via a topic/routing
  192. key (ie. kafka_avro_schema_topic).
  193. + print plugin: introduced new config directive avro_schema_output_file
  194. to save the Apache Avro schema in a separate file (it was only possible
  195. to have it combined at the beginning of the data file).
  196. + BGP daemon: introduced a new bgp_daemon_as config directive to set a
  197. LocalAS which could be different from the remote peer one. This is to
  198. establish an eBGP session instead of a iBGP one (default).
  199. + flow_to_rd_map: introduced support for mpls_vpn_id. In NetFlow/IPFIX
  200. this is compared against Field Types #234 and #235.
  201. + sfacctd: introduced support for sFlow v2/v4 counter samples (generic,
  202. ethernet, vlan). This is in addition to existing support for sFlow v5
  203. counters.
  204. + BGP, BMP and Streming Telemetry daemons: added writer_id field when
  205. writing to Kafka and/or RabbitMQ. The field reports the configured
  206. core_proc_name and the actual PID of the writer process (so, while
  207. being able to correlate writes to the same daemon, it's also possible
  208. to distinguish among overlapping writes).
  209. + amqp, kafka, print plugins: harmonized JSON output to the above: added
  210. event_type field, writer_id field with plugin name and PID.
  211. + BGP, BMP daemons: added AFI, SAFI information to log and dump outputs;
  212. also show VPN Label if SAFI is MPLS VPN.
  213. + pmbgpd, pmbmpd: added logics to bypass building RIBs if only logging
  214. BGP/BMP data real-time.
  215. + BMP daemon: added BMP peer TCP port to log and dump outputs (for NAT
  216. traversal scenarios). Contextually, multiple TCP sessions per IP are
  217. now supported for the same reason.
  218. + SQL plugins: ported (from print, etc. plugins) the 1.6.1 re-working of
  219. the max_writers feature.
  220. + uacctd: use current time when we don't have a timestamp from netlink.
  221. We only get a timestamp when there is a timestamp in the skb. Notably,
  222. locally generated packets don't get a timestamp. The patch is courtesy
  223. by Vincent Bernat ( @vincentbernat ).
  224. + build system: added configure options for partial linking of binaries
  225. with any selection/combination of IPv4/IPv6 accounting daemons, BGP
  226. daemon, BMP daemon and Streaming Telemetry daemon possible. By default
  227. all are compiled in.
  228. + BMP daemon: internal code changes to pass additional info from BMP
  229. per-peer header to bgp_parse_update_msg(). Goal is to expose further
  230. info, ie. pre- vs post- policy, when logging or dumping BMP info.
  231. ! fix, BGP daemon: introduced parsing of IPv6 MPLS VPN (vpnv6) NLRIs.
  232. Thanks to Alberto Santos ( @m4ccbr ) for reporting the issue.
  233. ! fix, BGP daemon: upon doing routes lookup, now correctly honouring
  234. the case of BGP-LU (SAFI_MPLS_LABEL).
  235. ! fix, BGP daemon: send BGP NOTIFICATION out in case of known failures
  236. in bgp_parse_msg().
  237. ! fix, kafka_partition, *_kafka_partition: default value changed from 0
  238. (partition zero) to -1 (RD_KAFKA_PARTITION_UA, partition unassigned).
  239. Thanks to Johan van den Dorpe ( @johanek ) for his support.
  240. ! fix, pre_tag_map: removed constraint for 'ip' keyword for nfacctd and
  241. sfacctd maps. While this is equivalent syntax to specifying rules with
  242. 'ip=', it allows for map indexing (maps_index: true).
  243. ! fix, bgp_agent_map: improved sanity check against bgp_ip for IPv6
  244. addresses (ie. an issue appeared for the case of '::1' where the first
  245. 64 bits are zeroed out). Thanks to Charlie Smurthwaite ( @catphish )
  246. for reporting the issue.
  247. ! fix, maps_index: indexing now correctly works for IPv6 pre_tag_map
  248. entries. That is, those where 'ip', the IP address of the NetFlow/
  249. IPFIX/sFlow exporter, is an IPv6 address.
  250. ! fix, pre_tag_map: if mpls_vpn_rd matching condition is specified and
  251. maps_index is enabled, PT_map_index_fdata_mpls_vpn_rd_handler() now
  252. picks the right (and expected) info.
  253. ! fix, pkt_handlers.c: improved definition and condition to free() in
  254. bgp_ext_handler() in order to prevent SEGVs. Thanks to Paul Mabey for
  255. his support.
  256. ! fix, kafka_common.c: removed waiting time from p_kafka_set_topic().
  257. Added docs advicing to create in advance Kafka topics.
  258. ! fix, sfacctd, sfprobe: tag and tag2 are now correctly re-defined as
  259. 64 bits long.
  260. ! fix, sfprobe plugin, sfacctd: tags and class primitives are now being
  261. encoded/decoded using enterprise #43874, legit, instead of #8800, that
  262. was squatted back in the times. See issue #71 on GiHub for more info.
  263. ! fix, sfacctd: lengthCheck() + skipBytes() were producing an incorrect
  264. jump in case of unknown flow samples. Replaced by skipBytesAndCheck().
  265. Thanks to Elisa Jasinska ( @fooelisa ) for her support.
  266. ! fix, pretag_handlers.c: in bgp_agent_map added case for 'vlan and ...'
  267. filter values.
  268. ! fix, BGP daemon: multiple issues of partial visibility of the stored
  269. RIBs and SEGVs when bgp_table_per_peer_buckets was not left default:
  270. don't mess with bms->table_per_peer_buckets given the multi-threaded
  271. scenario. Thanks to Dan Berger ( @dfberger ) for his support.
  272. ! fix, BGP, BMP daemons: bgp_process_withdraw() function init aligned to
  273. bgp_process_update() in order to prevent SEGVs. Thanks to Yuri Lachin
  274. for his support.
  275. ! fix, bgp_msg.c: Route Distinguisher was stored and printed incorrectly
  276. when of type RD_TYPE_IP. Thanks to Alberto Santos ( @m4ccbr ) for
  277. reporting the issue.
  278. ! fix, bgp_logdump.c: p_kafka_set_topic() was being wrongly applied to
  279. an amqp_host structure (instead of a kafka_host structure). Thanks to
  280. Corentin Neau ( @weyfonk ) for reporting the issue.
  281. ! fix, BGP daemon: improved BGP next-hop setting and comparison in cases
  282. of MP_REACH_NLRI and MPLS VPNs. Many thanks to both Catalin Petrescu
  283. ( @cpmarvin ) and Alberto Santos ( @m4ccbr ) for their support.
  284. ! fix, pmbgpd, pmbmpd: pidfile was not written even if configured. Thanks
  285. to Aaron Glenn ( @aaglenn ) for reporting the issue.
  286. ! fix, tee plugin: tee_max_receiver_pools is now correctly honoured and
  287. debug message shows the replicatd protocol, ie. NetFlow/IPFIX vs sFlow.
  288. ! AMQP, Kafka plugins: separate JSON objects, newline separated, are
  289. preferred to JSON arrays when buffering of output is enabled (ie.
  290. kafka_multi_values) and output is set to JSON. This is due to quicker
  291. serialisation performance shown by the Jansson library.
  292. ! build system: switched to enable IPv6 support by default (while the
  293. --disable-ipv6 knob can be used to reverse the behaviour). Patch is
  294. courtesy by Elisa Jasinska ( @fooelisa ).
  295. ! build system: given visibility, ie. via -V CL option, into compile
  296. options enabled by default (ie. IPv6, threads, 64bit counters, etc.).
  297. ! fix, nfprobe: free expired records when exporting to an unavailable
  298. collector in order to prevent a memory leak. Patch is courtersy by
  299. Vladimir Kunschikov ( @kunschikov ).
  300. ! fix, AMQP plugin: set content type to binary in case of Apache Avro
  301. output.
  302. ! fix, AMQP, Kafka plugins: optimized amqp_avro_schema_routing_key and
  303. kafka_avro_schema_topic. Avro schema is built only once at startup.
  304. ! fix, cfg.c: improved parsing of config key-values where squared brakets
  305. appear in the value part. Thanks to Brad Hein ( @regulatre ) for
  306. reporting the issue. Also, detection of duplicates among plugin and
  307. core process names was improved.
  308. ! fix, misc: compiler warnings: fix up missing includes and prototypes;
  309. the patch is courtesy by Tim LaBerge ( @tlaberge ).
  310. !, Kafka, RabbitMQ consumer example
  311. scripts have been greatly expanded to support posting to a REST API or
  312. to a new Kafka topic, including some stats. Also conversion of multiple
  313. newline-separated JSON objects to a JSON array has been added. Misc
  314. bugs were fixed.
  315. 1.6.1 -- 31-10-2016
  316. + Introduced pmbgpd daemon: a stand-alone BGP collector daemon; acts as a
  317. passive neighbor and maintains per-peer RIBs; can log real-time and/or
  318. dump at regular time-intervals BGP data to configured backends.
  319. + Introduced pmbmpd daemon: a stand-alone BMP collector daemon; can log
  320. real-time and/or dump at regular time-intervals BMP and BGP data to
  321. configured backends.
  322. + Introduced Apache Avro as part of print, AMQP and Kafka output: Apache
  323. Avro is a data serialization system providing rich data structures, a
  324. compact, fast, binary data format, a container file to store persistent
  325. data, remote procedure call (RPC) and simple integration with dynamic
  326. languages. The implementation is courtesy by Codethink Ltd.
  327. + as_path, std_comm and ext_comm primitives: along with their src counter
  328. parts, ie. src_as_path etc., have been re-worked to a variagle-length
  329. internal representation which will lead, when using BGP primitives, to
  330. memory savings of up to 50% compared to previous releases.
  331. + std_comm, ext_comm primitives: primitives are de-coupled so that they
  332. are not multiplexed anymore in the same field, on output. Added a
  333. tmp_comms_same_field config directive for backward compatibility.
  334. + nfacctd: added support for repeated NetFlow v9/IPFIX field types. Also
  335. flowStartDeltaMicroseconds (IE #158) and flowEndDeltaMicroseconds (#159)
  336. are now supported for timestamping.
  337. + kafka plugin: it is now possible to specify -1 (RD_KAFKA_RTITION_UA) as
  338. part of the kafka_partition config directive. Also, introduced support
  339. for Kafka partition keys via kafka_partition_key and equivalent config
  340. directives.
  341. + kafka plugin: kafka_broker_host directive now allows to specify multiple
  342. brokers, ie. "broker1:10000,broker2". The feature relies on capabilities
  343. of underlying rd_kafka_brokers_add().
  344. + tee, nfprobe, sfprobe plugins: introduced Kafka support for internal
  345. pipe and buffering, ie. plugin_pipe_kafka. This is in addition to the
  346. existing support for homegrown internal buffering and RabbitMQ.
  347. + tee plugin: introduced support for variable-length buffers which reduces
  348. CPU utilization.
  349. + print, MongoDB, AMQP and Kafka plugins: re-worked max_writers feature to
  350. not rely anymore on waitpid() inside signal handlers as it was failing on
  351. some OS versions (and could not be reproduced on others). Thanks to
  352. Janet Sullivan for her support.
  353. + bgp_follow_nexthop_external: introduced feature to return, when true, the
  354. next-hop from the routing table of the last node part of the supplied IP
  355. prefix(es) as value for the 'peer_ip_dst' primitive. When false, default,
  356. it returns the IP address of the last node part of the bgp_follow_nexthop
  357. config key.
  358. + pmtelemetryd: added initial support for GPB. Input GPB data is currently
  359. base64'd in the telemetry_data field of the daemon output JSON object.
  360. + pmtelemetryd: Added telemetry statistics. For each peer, track the number
  361. of packets received, how many bytes are pulled off the wire, and the
  362. resulting message payload. Dump these counts in logdump. Patch is courtesy
  363. by Tim LaBerge.
  364. + amqp_markers, kafka_markers: added start/end markers feature to AMQP and
  365. Kafka plugins output same as for the print plugin (print_markers).
  366. + pre_tag_map: 'direction' keyword now applies to sFlow too: it does expect
  367. values 0 (ingress direction) or 1 (egress direction), just like before.
  368. In sFlow v2/v4/v5 this returns a positive match if: 1) source_id equals
  369. to input interface and this 'direction' key is set to '0' or 2) source_id
  370. equals to output interface and this 'direction' key is set to '1'.
  371. + bgp_agent_map: introduced support for input and output interfaces. This
  372. is relevant to VPN scenarios.
  373. + tmp_asa_bi_flow hack: bi-flows use two counters to report counters, ie.
  374. bytes and packets, in forward and reverse directions. This hack (ab)uses
  375. the packets field in order to store the extra bytes counter.
  376. ! fix, nfacctd: debugging NetFlow v9/IPFIX templates, added original field
  377. type number to the output when the field is known and its description is
  378. presented.
  379. ! fix, Jansson: added JSON_PRESERVE_ORDER flag to json_dumps() to give
  380. output consistency across runs.
  381. ! fix, kafka_common.c: added rd_kafka_message_destroy() to p_kafka_consume_
  382. _data() to prevent memory leaks. Thanks to Paul Mabey for his support
  383. solving the issue.
  384. ! fix, kafka_common.c: p_kafka_set_topic() now gives it some time for the
  385. topic to get (auto) created, if needed.
  386. ! fix, print plugin: improved check for when to print table title (csv,
  387. formatted). Either 1) print_output_file_append is set to false or 2)
  388. print_output_file_append is set to true and file is to be created.
  389. ! fix, print_markers: start marker is now printed also in the case where
  390. print_output_file_append is set to true. Also, markers are now printed as
  391. a JSON object, if output is set to JSON.
  392. ! fix, pkt_handlers.c: removed l3_proto checks from NF_peer_dst_ip_handler()
  393. for cases where a v6 flows has a v4 BGP next-hop (ie. vpnv6)
  394. ! fix, pre_tag_map: removed 32 chars length limit from set_label statement.
  395. ! fix, custom primitives: names are now interpreted as case-insensitive.
  396. Patch is courtesy by Corentin Neau.
  397. ! fix, BGP, BMP and Streaming Telemetry: if reopening [bgp, bmp, telemetry]_
  398. daemon_msglog_file via SIGHUP, reset reload flag.
  399. ! fix, BGP, BMP and Streaming Telemetry: removed gettimeofday() from bgp_
  400. peer_dump_init() and bgp_peer_dump_close() in order to maintain a single
  401. timestamp for a full dump event. Thanks to Tim LaBerge for his support.
  402. ! fix, BGP, BMP and Streaming Telemetry: output log and dump messages went
  403. through a general review to improve information consistency and usability.
  404. Message formats are now documented in docs/MSGLOG_DUMP_FORMATS so to more
  405. easily track future changes.
  406. ! fix, pmtelemetryd: avoiding un-necessary spawn of a default plugin if none
  407. is defined.
  408. ! fix, pmtelemetryd: Mask SIGCHLD during socket IO. If we happen to be
  409. blocked in recv() while a log dump happens, recv() will fail with EINTR.
  410. This is to mask SIGCHLD during socket IO and restores the original mask
  411. after the IO completes. Patch is courtesy by Tim LaBerge.
  412. ! fix, build system: misc improvements made to the build system introduced
  413. in 1.6.0. Thanks to Vincent Bernat for his support in this area.
  414. ! fix, compiler warnings: ongoing effort to suppress warning messages when
  415. compiling. Thanks to Tim LaBerge, Matin Mitchell for their contributions.
  416. 1.6.0 -- 07-06-2016
  417. + Streaming telemetry daemon: quoting Cisco IOS-XR Telemetry Configuration
  418. Guide at the time of this writing: "Streaming telemetry [ .. ] data
  419. can be used for analysis and troubleshooting purposes to maintain the
  420. health of the network. This is achieved by leveraging the capabilities of
  421. machine-to-machine communication. [ .. ]" Streming telemetry support comes
  422. in two flavours: 1) a telemetry thread can be started in existing daemons,
  423. ie. sFlow, NetFlow/IPFIX, etc. for the purpose of data correlation and 2)
  424. a new daemon pmtelemetryd for standalone consumpton of data. Streaming
  425. network telemetry data can be logged real-time and/or dumped at regular
  426. time intervals to flat-files, RabbitMQ or Kafka brokers.
  427. + BMP daemon: introduced support for Route Monitoring messages. RM messages
  428. "provide an initial dump of all routes received from a peer as well as an
  429. ongoing mechanism that sends the incremental routes advertised and
  430. withdrawn by a peer to the monitoring station". Like for BMP events, RM
  431. messages can be logged real-time and/or dumped at regular time intervals
  432. to flat-files, RabbiMQ and Kafka brokers. RM messages are also saved in a
  433. RIB structure for IP prefix lookup.
  434. + uacctd: ULOG support switched to NFLOG, the newer and L3 independent Linux
  435. packet logging framework. One of the key advantages of NFLOG is support for
  436. IPv4 and IPv6 (whereas ULOG was restricted to IPv4 only). The code has been
  437. contributed by Vincent Bernat ( @vincentbernat ).
  438. + build system: it was modernized so not to rely on specific and old versions
  439. of automake and autoconf, as it was the case until 1.5. Among the things,
  440. pkg-config and libtool are leveraged and an script is generated.
  441. The code has been contributed by Vincent Bernat ( @vincentbernat ).
  442. + sfacctd: RabbitMQ and Kafka support was introduced to real-time log and/
  443. or dump at regular time intervals of sFlow counters. This is in addition
  444. to existing support for flat-files.
  445. + maps_index: several improvements were carried out in the area of indexing
  446. of maps: optimizations to pretag_index_fill() and pretag_index_lookup() to
  447. improve lookup speeds; optimized id_entry structure, ie. by splitting key
  448. and non-key parts, and hashing key in order to consume less memory; added
  449. duplicate entry detection (cause of sudden index destruction);
  450. pretag_index_destroy() destroys hash keys for each index entry, solving a
  451. memory leak issue. Thanks to Job Snijders ( @job ) for his support.
  452. + Introduced 'export_proto_seqno' aggregation primitive to report on
  453. sequence number of the export protocol (ie. NetFlow, sFlow, IPFIX). This
  454. feature may enable more advanced offline analysis of packet loss, out of
  455. orders, etc. over time windows than basic online analytics provided by the
  456. daemons.
  457. + log.c: logging moved from standard output (stdout) to standard error
  458. (stderr) so to not conflict with stdout printing of statistics (print
  459. plugin). Thanks to Jim Westfall ( @jwestfall69 ) for his support.
  460. + print plugin: introduced a new print_output_lock_file config directive
  461. to lock standard output (stdout) output so to prevent multiple processes
  462. (instances of the same print plugin or different instances of print plugin)
  463. overlap output. Thanks to Jim Westfall ( @jwestfall69 ) for his support.
  464. + pkt_handlers.c: euristics in NetFlow v9/IPFIX VLAN handler were improved
  465. for the case of flows in egress direction. Also IP protocol checks were
  466. removed for UDP/TCP ports and TCP flags in case the export protocol is
  467. NetFlow v9/IPFIX. Thanks to Alexander Ponamarchuk for his support.
  468. ! Code refactoring: improved re-usability of much of the BGP code (so to
  469. make it possible to use it as a library for some BMP daemon features, ie.
  470. Route Monitoring messages support); consolidated functions to handle log
  471. and print plugin output files; improved log messages to always include
  472. process name and type.
  473. ! fix, bpf_filter.c: issue compiling against libpcap 1.7.x; introduced a
  474. check for existing bpf_filter() in libpcap in order to prevent namespace
  475. conflicts.
  476. ! fix, tmp_net_own_field default value changed to true. This knob can be
  477. still switched to false for this release but is going to be removed soon.
  478. ! fix, cfg.c, cfg_handlers.c, pmacct.c: some configuration directives and
  479. pmacct CL parameters requiring string parsing, ie. -T -O -c, are now
  480. passed through tolower().
  481. ! fix, MongoDB plugin: removed version check around mongo_create_index()
  482. and now defaulting to latest MongoDB C legacy driver API. This is due to
  483. some versioning issue in the driver.
  484. ! fix, timestamp_arrival: primitive was reporting incorrect results (ie.
  485. always zero) if timestamp_start or timestamp_end were not also specified
  486. as part of the same aggregation method. Many thanks to Vincent Morel for
  487. reporting the issue.
  488. ! fix, thread stack: a value of 0, default, leaves the stack size to the
  489. system default or pmacct minimum (8192000) if system default is too low.
  490. Some systems may throw an error if the defined size is not a multiple of
  491. the system page size.
  492. ! fix, nfacctd: improved NetFlow v9/IPFIX parsing. Added new length checks
  493. and fixed some existing checks. Thanks to Robert Wuttke ( @Benocs ) for his
  494. support.
  495. ! fix, pretag_handlers.c: BPAS_map_bgp_nexthop_handler() and BPAS_map_bgp_
  496. peer_dst_as_handler() were not setting a func_type.
  497. ! fix, JSON support: Jansson 2.2 does not have json_object_update_missing()
  498. function which was introduced in 2.3. This is not provided as part of a
  499. jansson.c file and compiled in conditionally, if needed. Jansson 2.2 is
  500. still shipped along by some recent OS releases. Thanks to Vincent Bernat
  501. ( @vincentbernat ) for contributing the patch.
  502. ! fix, log.c: use a format string when calling syslog(). Passing directly a
  503. potentially uncontrolled string could crash the program if the string
  504. contains formatting parameters. Thanks to Vincent Bernat ( @vincentbernat )
  505. for contributing the patch.
  506. ! fix, sfacctd.c: default value for config.sfacctd_counter_max_nodes was set
  507. after sf_cnt_link_misc_structs(). Thanks to Robin Douine for his support
  508. resolving the issue.
  509. ! fix, sfacctd.c: timestamp was consistently being reported as null in sFlow
  510. counters output. Thanks to Robin Douine for his support resolving the issue.
  511. ! fix, SQL plugins: $SQL_HISTORY_BASETIME environment variable was reporting a
  512. wrong value (next basetime) in the sql_trigger_exec script. Thanks to Rain
  513. Nõmm for reporting the issue.
  514. ! fix, pretag.c: in pretag_index_fill(), replaced memcpy() with hash_dup_key()
  515. also a missing res_fdata initialization in pretag_index_lookup() was solved;
  516. these issues were originating false negatives upon lookup. Thanks to Rain
  517. Nõmm fo his suppor.
  518. ! fix, ISIS daemon: hash_* functions renamed into isis_hash_* to avoid name
  519. space clashes with their BGP daemon counter-parts.
  520. ! fix, kafka_common.c: rd_kafka_conf_set_log_cb moved to p_kafka_init_host()
  521. due to crashes seen in p_kafka_connect_to_produce(). Thanks to Paul Mabey
  522. for his support resolving the issue.
  523. ! fix, bgp_lookup.c: bgp_node_match_* were not returning any match in
  524. bgp_follow_nexthop_lookup(). Thanks to Tim Jackson ( @jackson-tim ) for his
  525. support resolving the issue.
  526. ! fix, sql_common.c: crashes observed when nfacctd_stitching was set to true
  527. and nfacctd_time_new was set to false. Thanks to Jaroslav Jiráse
  528. ( @jjirasek ) for his support solving the issue.
  529. - SQL plugins: sql_recovery_logfile feature was removed from the code due
  530. to lack of support and interest. Along with it, also pmmyplay and pmpgplay
  531. tools have been removed.
  532. - pre_tag_map: removed support for mpls_pw_id due to lack of interest.
  533. 1.5.3 -- 14-01-2016
  534. + Introduced the Kafka plugin: Apache Kafka is publish-subscribe messaging
  535. rethought as a distributed commit log. Its qualities being: fast, scalable,
  536. durable and distributed by design. pmacct Kafka plugin is designed to
  537. send aggregated network traffic data, in JSON format, through a Kafka
  538. broker to 3rd party applications.
  539. + Introduced Kafka support to BGP and BMP daemons, in both their msglog
  540. and dump flavors (ie. see [bgp|bmp]_daemon_msglog_kafka_broker_host and
  541. [bgp_table|bmp]_dump_kafka_broker_host and companion config directives).
  542. + Introduced support for a Kafka broker to be used for queueing and data
  543. exchange between Core Process and plugins. plugin_pipe_kafka directive,
  544. along with all other plugin_pipe_kafka_* directives, can be set globally
  545. or apply on a per plugin basis - similarly to what was done for RabbitMQ
  546. (ie. plugin_pipe_amqp). Support is currently restricted only to print
  547. plugin.
  548. + Added a new timestamp_arrival primitive to expose NetFlow/IPFIX records
  549. observation time (ie. arrival at the collector), in addition to flows
  550. start and end times (timestamp_start and timestamp_end respectively).
  551. + plugin_pipe_amqp: feature extended to the plugins missing it: nfprobe,
  552. sfprobe and tee.
  553. + Introduced bgp_table_dump_latest_file: defines the full pathname to
  554. pointer(s) to latest file(s). Update of the latest pointer is done
  555. evaluating files modification time. Many thanks to Juan Camilo Cardona
  556. ( @jccardonar ) for proposing the feature.
  557. + Introduced pmacctd_nonroot config directive to allow to run pmacctd
  558. from a user with non root privileges. This can be desirable on systems
  559. supporting a tool like setcap, ie. 'setcap "cap_net_raw,cap_net_admin=ep"
  560. /path/to/pmacctd', to assign specific system capabilities to unprivileged
  561. users. Patch is courtesy by Laurent Oudot ( @loudot-tehtris ).
  562. + Introduced plugin_pipe_check_core_pid: when enabled (default), validates
  563. the sender of data at the plugin side. Useful when plugin_pipe_amqp or
  564. plugin_pipe_kafka are enabled and hence a broker sits between the daemon
  565. Core Process and the Plugins.
  566. + A new debug_internal_msg config directive to specifically enable debug
  567. of internal messaging between Core process and plugins.
  568. ! bgp_table_dump_refresh_time, bmp_dump_refresh_time: max allowed value
  569. raised to 86400 from 3600.
  570. ! [n|s]facctd_as_new renamed [n|s]facctd_as; improved input checks to all
  571. *_as (ie. nfacctd_as) and *_net (ie. nfacctd_net) config directives.
  572. ! pkt_handlers.c: NF_sampling_rate_handler(), SF_sampling_rate_handler()
  573. now perform a renormalization check at last (instead of at first) so to
  574. report the case of unknown (0) sampling rate.
  575. ! plugin_pipe_amqp_routing_key: default value changed to '$core_proc_name-
  576. $plugin_name-$plugin_type'. Also, increased flexibility for customizing
  577. the key with the use of variables (values computed at startup).
  578. ! Improved example with CL arguments and better exception
  579. handling. Also removed file, example is now merged
  580. in
  581. ! fix, BGP daemon: several code optimizations and a few starving conditions
  582. fixed. Thanks to Markus Weber ( @FvDxxx ) for his peer index round-robin
  583. patch; thanks also to Job Snijders ( @job ) for his extensive support in
  584. this area.
  585. ! fix, BMP daemon: greatly improved message parsing and segment reassembly;
  586. RabbitMQ broker support found broken; several code optimizations are also
  587. included.
  588. ! fix, bgp_table.c: bgp_table_top(), added input check to prevent crashes
  589. in cases table contains no routes.
  590. ! fix, networks_file: missing atoi() for networks_cache_entries. Patch is
  591. courtesy by Markus Weber ( @FvDxxx ).
  592. ! fix, plugin_pipe_amqp_routing_key: check introduced to prevent multiple
  593. plugins to bind to the same RabbitMQ exchange, routing key combination.
  594. Thanks to Jerred Horsman for reporting the issue.
  595. ! fix, MongoDB plugin: added a custom oid fuzz generator to prevent
  596. concurrent inserts to fail; switched from deprecated mongo_connect() to
  597. mongo_client(); added MONGO_CONTINUE_ON_ERROR flag to mongo_insert_batch
  598. along with more verbose error reporting. Patches are all courtesy by
  599. Russell Heilling ( @xchewtoyx ).
  600. ! fix, nl.c: increments made too early after introduction of MAX_GTP_TRIALS
  601. Affected: pmacctd processing of GTP in releases 1.5.x. Patch is courtesy
  602. by TANAKA Masayuki ( @tanakamasayuki ).
  603. ! fix, pkt_handlers.c: improved case for no SAMPLER_ID, ALU & IPFIX in
  604. NF_sampling_rate_handler() on par with NF_counters_renormalize_handler().
  605. ! fix, SQL scripts: always use "DROP TABLE IF EXISTS" for both PostgreSQL
  606. and SQLite. Pathes are courtesy by Vincent Bernat ( @vincentbernat ).
  607. ! fix, plugin_hooks.c: if p_amqp_publish_binary() calls were done while a
  608. sleeper thread was launched, a memory corruption was observed.
  609. ! fix, util.c: mkdir() calls in mkdir_multilevel() now default to mode 777
  610. instead of 700; this allows more play with files_umask (by default 077).
  611. Thanks to Ruben Laban for reporting the issue.
  612. ! fix, BMP daemon: solved a build issue under MacOS X. Path is courtesy by
  613. Junpei YOSHINO ( @junpei-yoshino ).
  614. ! fix, util.c: self-defined Malloc() can allocate more than 4GB of memory;
  615. function is also now renamed pm_malloc().
  616. ! fix, PostgreSQL plugin: upon purge, call sql_query() only if status of
  617. the entry is SQL_CACHE_COMMITTED. Thanks to Harry Foster ( @harryfoster )
  618. for his support resolving the issue.
  619. ! fix, building system: link pfring before pcap to prevend failures when
  620. linking. Patch is courtesy by @matthewsf .
  621. ! fix, plugin_common.c: memory leak discovered when pending queries queue
  622. was involved (ie. cases where print_refresh_time > print_history). Thanks
  623. to Edward Henigin for reporting the issue.
  624. 1.5.2 -- 07-09-2015
  625. + Introduced support for a RabbitMQ broker to be used for queueing and
  626. data exchange between Core Process and plugins. This is in alternative to
  627. the home-grown circular queue implementation. plugin_pipe_amqp directive,
  628. along with all other plugin_pipe_amqp_* directives, can be set globally
  629. or apply on a per plugin basis (ie. it is a valid scenario, if multiple
  630. plugins are instantiated, that some make use of home-grown queueing,
  631. while others use RabbitMQ based queueing).
  632. + Introducting support for Maximind GeoIP v2 (libmaxminddb) library: if
  633. pmacct is compiled with --enable-geoipv2, this defines full pathname to
  634. a Maxmind GeoIP database v2 (libmaxminddb) Only the binary database
  635. format is supported (ie. it is not possible to load distinct CSVs for
  636. IPv4 and IPv6 addresses).
  637. + Introduced infrastructure for sFlow counters and support specifically for
  638. generic, ethernet and vlan counters. Counters are exported in JSON format
  639. to files, specified via sfacctd_counter_file. The supplied filename can
  640. contain as variable the sFlow agent IP address.
  641. + Introduced a new thread_stack config directive to allow to modify the
  642. thread stack size. Natanael Copa reported that some libc implementations,
  643. ie. musl libc, may set a stack size that is too small by default.
  644. + Introduced networks_file_no_lpm feature: it applies when aggregation
  645. method includes src_net and/or dst_net and nfacctd_net (or equivalents)
  646. and/or nfacctd_as_new (or equivalents) are set to longest (or fallback):
  647. an IP prefix defined as part of the supplied networks_file wins always,
  648. even if it's not longest.
  649. + tee plugin: added support for (non-)transparent IPv6 replication [further
  650. QA required]
  651. + plugin_common.c, sql_common.c: added log message to estimate base cache
  652. memory usage.
  653. + print, AMQP, MongoDB plugins; sfacctd, BGP, BMP daemons: introducing
  654. timestamps_since_epoch to write timestamps in 'since Epoch' format.
  655. + nfacctd: flow bytes counter can now be sourced via element ID #352
  656. (layer2OctetDeltaCount) in addition to element ID's already supported.
  657. Thanks to Jonathan Thorpe for his support.
  658. + Introducing proc_priority: redefines the process scheduling priority,
  659. equivalent to using the 'nice' tool. Each daemon process, ie. core,
  660. plugins, etc., can define a different priority.
  661. ! fix, BMP daemon: improved preliminar checks in bmp_log_msg() and added
  662. missing SIGHUP signal handling to reload bmp_daemon_msglog_file files.
  663. ! fix, bgp_logdump.c: under certain configuration conditions call to both
  664. write_and_free_json() and write_and_free_json_amqp() was leading to SEGV.
  665. Thanks to Yuriy Lachin for reporting the issue.
  666. ! fix, BGP daemon: improved BGP dump output: more accurate timestamping of
  667. dump_init, dump_close events. dump_close now mentions amount of entries
  668. and tables dumped. Thanks to Yuriy Lachin for brainstorming around this.
  669. ! fix, cfg.c: raised amount of allowed config lines from 256 to 8K.
  670. ! fix, print/AMQP/MongoDB plugins: SEGV observed when IPFIX vlen variables
  671. were stored in the pending_queries_queue structure (ie. as a result of a
  672. time mismatch among the IPFIX exporter and the collector box).
  673. ! fix, vlen primitives: when 'raw' semantics was selected, print_hex() was
  674. returning wrong hex string length (one char short). As a consequence
  675. occasionally some extra dirty chars were seen at the end of the converted
  676. string.
  677. ! fix, vlen primitives: memory leak verified in print/AMQP/MongoDB plugins.
  678. ! fix, print, MongoDB & AMQP plugins: dirty values printed as part of the
  679. 'proto' under certain conditions. Thanks to Rene Stoutjesdijk for his
  680. support resolving the issue.
  681. ! fix, amqp_common.c: amqp_exchange_declare() call changed so to address
  682. the change of rabbitmq-c API for support of auto_delete & internal for
  683. exchange.declare. Backward compatibility with rabbitmq-c <= 0.5.2 is
  684. also taken care of. Thanks to Brent Van Dussen for reporting the issue.
  685. ! fix, compiling on recent FreeBSD: solved some errors catched by the -Wall
  686. compiler flag. Thanks to Stephen Fulton for reporting the issue. Most of
  687. the patch is courtesy by Mike Bowie.
  688. ! fix, print/AMQP/MongoDB plugins: enforcing cleanup of malloc()ed structs
  689. part of entries added to the pending queue, ie. because seen as future
  690. entries due to a mismatch of the collector clock with the one of NetFlow/
  691. IPFIX exporter(s). This may have lead to data inconsistencies.
  692. ! fix, amqp_common.c: Content type was only specified for messages published
  693. when the amqp_persistent_msg configuration option is specified. This info
  694. should always be applied to describe the payload of the message. Patch is
  695. courtesy by Will Dowling.
  696. ! fix, amqp_plugin.c: generate an error on compile if --enable-rabbitmq is
  697. specified without --enable-jansson. It's clear in the documentation that
  698. both are required for AMQP support, but if built without jansson it will
  699. silently not publish messages to AMQP. Patch is courtesy by Will Dowling.
  700. ! fix, amqp_common.c: modified the content type to "application/json" in
  701. line with RFC4627. Patch is courtesy by Will Dowling.
  702. ! fix, setsockopt(): u_int64_t pipe_size vars changed to int, in line with
  703. typical OS buffer limits (Linux, Solaris). Introduced check supplied pipe
  704. size values are not bigger than INT_MAX. Many thanks to Markus Weber for
  705. reporting the issue.
  706. ! fix, nl.c: removed pretag_free_label() from pcap_cb() and ensuring init
  707. of pptrs. Under certain conditions SEGVs could be noticed.
  708. ! fix, flow stitching: when print/AMQP/MongoDB plugins were making use of
  709. the pending queries queue, ie. to compensate for time offsets/flows in
  710. the future, the stitching feature could potentially lead to SEGV due to
  711. unsettled pointers.
  712. ! fix, pgsql plugin: SEGV were noticed when insert/update queries to the
  713. PostgreSQL database were returning different than PGRES_COMMAND_OK, hence
  714. triggering the reprocess mechanism. Thanks very much to Alan Turower for
  715. his support.
  716. ! fix, improved logging of elements received/sent at buffering point between
  717. core process and plugins. Also added explicit start/end purge log message
  718. for cases in which there is no data to purge.
  719. ! fix, signals.c: ignore_falling_child() now logs if a child process exited
  720. with abnormal conditions; this is useful to track writer processes (created
  721. by plugins) are terminated by a signal, ie. SEGV. This is already the case
  722. for plugins themselves, with the Core Process reporting a simlar log
  723. message in case of abnormal exit. Thanks very much to Rene Stoutjesdijk
  724. for his support.
  725. ! fix, preprocess-data.h: added supported functions minf, minb, minbpp and
  726. minppf to non SQL plugins. Thanks to Jared Deyo for reporting the issue.
  727. ! fix, nfprobe_plugin.c: IP protocol was not set up correctly for IPv6
  728. traffic in NetFlow v9/IPFIX. Thanks to Gabriel Vermeulen his support
  729. solving the issue.
  730. 1.5.1 -- 21-02-2015
  731. + BMP daemon: BMP, BGP Monitoring Protocol, can be used to monitor BGP
  732. sessions. The current implementation is base on the draft-ietf-grow-bmp-07
  733. IETF draft. The daemon currently supports BMP events and stats only, ie.
  734. initiation, termination, peer up, peer down and stats reports messages.
  735. Route Monitoring is future (upcoming) work but routes can be currently
  736. sourced via the BGP daemon thread (best path only or ADD-PATH), making
  737. the two daemons complementary. The daemon enables to write BMP messages
  738. to files or AMQP queues, real-time (msglog) or at regular time intervals
  739. (dump) and is a separate thread in the NetFlow (nfacctd) or sFlow (sfacctd)
  740. collectors.
  741. + tmp_net_own_field directive is introduced to record both individual source
  742. and destination IP addresses and their IP prefix (nets) as part of the same
  743. aggregation method. While this should become default behaviour, a knob for
  744. backward-compatibility is made available for all 1.5 until the next major
  745. release.
  746. + Introduced nfacctd_stitching and equivalents (ie. sfacctd_stitching):
  747. when set to true, given an aggregation method, two new non-key fields are
  748. added to the aggregate upon purging data to the backend: timestamp_min is
  749. the timestamp of the first element contributing to a certain aggregate
  750. and timestamp_max is the timestamp of the last element. In case the export
  751. protocol provides time references, ie. NetFlow/IPFIX, these are used; if not
  752. the current time (hence time of arrival to the collector) is used instead.
  753. + Introduced amqp_routing_key_rr feature to perform round-robin load-
  754. balancing over a set of routing keys. This is in addition to existing,
  755. and more involved, functionality of tag-based load-balancing.
  756. + Introduced amqp_multi_values feature: this is same feature in concept as
  757. sql_multi_values (see docs). The value is the amount of elements to pack
  758. in each JSON array.
  759. + Introduced amqp_vhost and companion (ie. bgp_daemon_msglog_amqp_vhost)
  760. configuration directives to define the AMQP/RabbitMQ server virtual host.
  761. + BGP daemon: bgp_daemon_id now allows to define the BGP Router-ID disjoint
  762. from the bgp_daemon_ip definition. Thanks to Bela Toros for his patch.
  763. + tee plugin: introduced tee_ipprec feature to color replicated packets,
  764. both in transparent and non-transparent modes. Useful, especially when
  765. in transparent mode and replicating to hosts in different subnets, to
  766. verify which packets are coming from the replicator.
  767. + tee plugin: plugin-kernel send buffer size is now configurable via a new
  768. config directive tee_pipe_size. Improved logging of send() failures.
  769. + nfacctd: introduced support for IPFIX sampling/renormalization using
  770. element IDs: #302 (selectorId), #305 (samplingPacketInterval) and #306
  771. (samplingPacketSpace). Many thanks to Rene Stoutjesdijk for his support.
  772. + nfacctd: added also support for VLAN ID for NetFlow v9/IPFIX via element
  773. type #243 (it was already supported via elements #58 and #59). Support was
  774. also added for 802.1p/CoS via element #244.
  775. + nfacctd: added native support for NetFlow v9/IPFIX IE #252 and #253 as
  776. part of existing primitives in_iface and out_iface (additional check).
  777. + pre_tag_map: introduced 'cvlan primitive. In NetFlow v9 and IPFIX this is
  778. compared against IE #245. The primitive also supports map indexing.
  779. + Introduced pre_tag_label_filter to filter on the 'label' primitive in a
  780. similar way how the existing pre_tag_filter feature works against the
  781. 'tag' primitive. Null label values (ie. unlabelled data) can be matched
  782. using the 'null' keyword. Negations are allowed by pre-pending a minus
  783. sign to the label value.
  784. + IMT plugin: introduced '-i' command-line option to pmacct client tool: it
  785. shows last time (in seconds) statistis were cleared via 'pmacct -e'.
  786. + print, MongoDB & AMQP plugins: sql_startup_delay feature ported to these
  787. plugins.
  788. ! sql_num_hosts: the feature has been improved to support IPv6 addresses.
  789. Pre-requisite is definition of INET6_ATON() function in the RDBMS, which
  790. is the case for MySQL >= 5.6.3. In SQLite such function has to be defined
  791. manually.
  792. ! nfacctd: improved NF_evaluate_flow_type() euristics to reckon NetFlow/
  793. IPFIX event (NAT, Firewall, etc.) vs traffic (flows) records.
  794. ! fix, GeoIP: spit log notification (warning) in case GeoIP_open() returns
  795. null pointer.
  796. ! fix, IMT plugin: pmacct client -M and -N queries were failing to report
  797. results on exact matches. Affected: 1.5.0. Thanks to Xavier Vitard for
  798. reporting the issue.
  799. ! fix, pkt_handlers.c: missing else in NF_src_host_handler() was causing
  800. IPv6 prefix being copied instead of IPv6 address against NetFlow v9 recs
  801. containing both info.
  802. ! fix, uacctd: informational log message now shows the correct group the
  803. daemon is bound to. Thanks to Marco Marzetti for reporting the issue.
  804. ! fix, nfv9_template.c: missing byte conversion while decoding templates
  805. was causing SEGV under certain conditions. Thanks to Sergio Bellini for
  806. reporting the issue.
  807. 1.5.0 -- 28-08-2014
  808. + Introduced bgp_daemon_msglog_file config directive to enable streamed
  809. logging of BGP messages/events. Each log entry features a time reference,
  810. BGP peer IP address, event type and a sequence number (to order events
  811. when time reference is not granular enough). BGP UPDATE messages also
  812. contain full prefix and BGP attributes information. Example given in
  813. QUICKSTART file, chapter XIIf.
  814. + Introduced dump of BGP tables at regular time intervals. The filename,
  815. which can include variables, is set by bgp_table_dump_file directive.
  816. The output format, currently only JSON, can be set in future via the
  817. bgp_table_dump_output directive. The time interval between dumps can
  818. be set via the bgp_table_dump_refresh_time directive. Example given in
  819. QUICKSTART file, chapter XIIf.
  820. + Introduced support for internally variable-length primitives (likely
  821. candidates are strings). Introduced also the 'label' primitive which
  822. is a variable-length string equivalent of tag and tag2 primitives. Its
  823. value are set via a 'set_label' statement in a pre_tag_map (see examples/
  824. If, ie. as a result of JEQ's in a pre_tag_map,
  825. multiple 'set_label' are applied, then default operation is append
  826. labels and separate by a comma.
  827. + pmacct project has been assigned PEN #43874. nfprobe plugin: tag, tag2,
  828. label primitives are now encoded in IPFIX making use of the pmacct PEN.
  829. + Ported preprocess feature to print, MongoDB and AMQP plugins. Preprocess
  830. allows to process aggregates (via a comma-separated list of conditionals
  831. and checks) while purging data to the backend thus resulting in a
  832. powerful selection tier. minp, minb, minf, minbpp, minppf checks have
  833. been currently ported. As a result of the porting a new set of config
  834. directives are added, ie. print_preprocess and print_preprocess_type.
  835. + print, MongoDB & AMQP plugins: if data (start/base) time is greater than
  836. commit time then place in pending queue and after purging event re-insert
  837. in cache. Concept ported from SQL plugins.
  838. + MySQL, PostgreSQL plugins: sql_locking_style now supports keyword "none"
  839. to disable locking. This method can help in certain cases, for example
  840. when grants over the whole database (requirement for "table" locking in
  841. MySQL) is not available.
  842. + util.c: open_logfile() now calls mkdir_multilevel() to allow building
  843. intermediate directory levels, if not existing. This brings all log
  844. files in line with capabilities of print_output_file directive.
  845. + Introduced [u|pm]acctd_flow_tcp_lifetime to defines how long a TCP flow
  846. could remain inactive. This is in addition to [u|pm]acctd_flow_lifetime
  847. that allows to define the same for generic, ie. non-TCP, flows. Thanks to
  848. Stathis Gkotsis for his support.
  849. + Introducing nfacctd_account_options: if set to true account for NetFlow/
  850. IPFIX option records as well as flow ones. pre_tag_map offers sample_type
  851. value of 'option' now to split option data records from flow ones.
  852. + nfprobe plugin: support for custom-defined primitives has been introduced
  853. in line with other plugins. With such feature it will be possible to
  854. augment NetFlow v9/IPFIX records with custom fields (in IPFIX also PENs
  855. are supported).
  856. + Built a minimal API, for internal use only, around AMQP. Goal is to make
  857. re-use of the same AMQP structures for different purposes (logging, BGP
  858. daemon dumps, AMQP plugin, etc.).
  859. ! fix, BGP daemon: introduced bgp_peer_info_delete() to delete/free BGP
  860. info after a BGP peer disconnects.
  861. ! fix, print, AMQP, memory plguins: when selecting JSON output, jansson
  862. library json_decref() is used in place of free() to free up memory
  863. allocated by JSON objects. Using free() was originating memory leaks.
  864. ! fix, AMQP plugin: in line with other plugins QN (query number or in case
  865. of AMQP messagess number) in log messages now reflects the real number of
  866. messages sent to the RabbitMQ message exchange and not just all messages
  867. in the queue. Thanks to Gabriel Snook for reporting the issue.
  868. ! fix, IMT plugin: memory leak due to missed calls to free_extra_allocs()
  869. in case all extras.off_* were null. Thanks to Tim Jackson for his support
  870. resolving the issue.
  871. ! fix, pmacctd: if reading from a pcap_savefile, introduce a short usleep()
  872. after each buffer worth of data so to give time plugins to process/cache
  873. it.
  874. ! fix, SQL plugins: SQL handler types now include primitives registry index
  875. ! fix, print, AMQP & MongoDB plugins: added free() for empty_pcust allocs
  876. ! fix, plugin hooks: improved checks to prevent the last buffer on a pipe
  877. to plugins (plugin_pipe_size) could go partly out of bounds.
  878. ! fix, nfacctd: improved handling of IPFIX vlen records.
  879. ! fix, nfprobe: SEGV if custom primitives are defined but array structure
  880. is not allocated.
  881. ! fix, nfprobe: wrong length was calculated in IPv6 templates for fields
  882. with PEN != 0.
  883. ! fix, plugin_common.c: declared struct pkt_data in P_cache_insert_pending
  884. to be pointed by prim_ptrs. primptrs_set_all_from_chained_cache() is now
  885. safe if prim_ptrs is null.
  886. ! fix, nfprobe: tackled the case of coexisting 1) PEN and non-PEN custom
  887. primitives and 2) variable and fixed custom primitives.
  888. ! fix, plugin_common.c: declared struct pkt_data in P_cache_insert_pending
  889. to be pointed by prim_ptrs. primptrs_set_all_from_chained_cache() is now
  890. safe if prim_ptrs is null.
  891. ! fix, lofging: selected configuration file is now logged. cfg_file is passed
  892. through realpath() in order to always log the absolute path.
  893. ! fix, print, MongoDB & AMQP plugins: pm_setproctitle() invoked upon forking
  894. writer processes in alignment with SQL plugins.
  895. ! fix, pmacct client: it's now possible to query and wildcard on primitives
  896. internally allocated over what_to_count_2 registry.
  897. 1.5.0rc3 -- 18-04-2014
  898. + BGP daemon: support for BGP ADD-PATH capability draft-ietf-idr-add-paths
  899. has been introduced, useful to advertise known paths when BGP multi-path
  900. is enabled in a network. The correct BGP info is linked to traffic data
  901. using BGP next-hop (or IP next-hop if use_ip_next_hop is set to true) as
  902. selector among the paths available.
  903. + pre_tag_map: de-globalized the feature so that, while Pre-Tagging is
  904. evaluated in the Core Process, each plugin can be defined a own/local
  905. pre_tag_map.
  906. + maps_row_len: directive introduced to define the maximum length of map
  907. (ie. pre_tag_map) rows. The default value is suitable for most scenarios,
  908. though tuning it could be required either to save on memory or to allow
  909. for longer entries (ie. filters).
  910. + Introduced use_ip_next_hop config directive: when IP prefix aggregation
  911. (ie. nfacctd_net) is set to 'netflow', 'sflow' or 'fallback' populate
  912. 'peer_dst_ip' field from NetFlow/sFlow IP next hop field if BGP next-hop
  913. is not available.
  914. + AMQP plugin: implemented persistent messaging via amqp_persistent_msg
  915. configuration directive so to protect against RabbitMQ restarts. Feature
  916. is courtesy by Nick Douma.
  917. + pmacct in-memory plugin client: -T option now supports how many entries
  918. to show via '<bytes | packets | flows>,[<# how many>]' argument syntax.
  919. + nfprobe plugin: take BGP next-hop from a defined networks_file. This is
  920. in addition to existing feature to take BGP next-hop from a BGP feed.
  921. + Set of *_proc_name configuration directives renamed to core_proc_name.
  922. Value of core_proc_name is now applied to logging functions and process
  923. title.
  924. + Re-implemented reverse BGP lookup based primitives, src_as_path src_med
  925. src_std_comm src_ext_comm and src_local_pref, in print, MongoDB and
  926. AMQP plugins. Primitives have also been re-documented.
  927. + pre_tag_map: set_tag and set_tag2 can now be auto-increasing values, ie.
  928. "set_tag=1++": "1" being the selected floor value at startup and "++"
  929. instructs to increase the tag value at every pre_tag_map iteration. Many
  930. thanks to Brent Van Dussen and Gabriel Snook for their support.
  931. + Added support for NetFlow v9/IPFIX source/destination IPv4/IPv6 prefixes
  932. encoded as flow types: #44, #45, #169 and #170.
  933. + [sql|print|mongo|amqp]_history and sql_trigger_time can now be specified
  934. also in seconds, ie. as '300' or '300s' alternatively to '5m'. This is to
  935. ease syncronization of these values against refresh time to the backend,
  936. ie. sql_refresh_time.
  937. + Added post_tag2 configuration directive to set tag2 similarly to what
  938. post_tag does.
  939. + SQL plugins: agent_id, agent_id2 fields renamed to tag, tag2. Issued SQL
  940. table schema #9 for agent_id backward compatibility. Renaming agent_id2
  941. to tag2 is going to be disruptive to existing deployments instead.
  942. UPGRADE doc updated.
  943. + print, MongoDB, AMQP plugins: added [print|mongo|amqp]_max_writers set of
  944. configuration directives to port from SQL plugins the idea of max number
  945. of concurrent writer processes the plugin is allowed to start.
  946. + util.c: comments can now start with a '#' symbol in addition to existing
  947. '!'.
  948. ! fix, BGP daemon: removed a non-contextual BGP message length check. Same
  949. check is already done in the part handling payload reassembly.
  950. ! fix, BGP daemon: MP_REACH_NLRI not assumed to be anymore at the end of a
  951. route announcement.
  952. ! fix, MySQL plugin: added linking of pmacct code against -lstdc++ and
  953. -lrt if MySQL plugin is enabled, pre-requisite for MySQL 5.6. Many
  954. thanks to Stefano Birmani for reporting the issue.
  955. ! fix, sql_common.c: memory leak affecting AS-PATH and BGP communities.
  956. Version 1.5.0rc2 affected. Thanks to Brent Van Dussen for his support
  957. solving the issue.
  958. ! fix, MongoDB plugin: timestamp_start, timestamp_end moved from timestamp
  959. type, reserved for internal use, to date.
  960. ! fix, print, memory, MongoDB, AMQP plugins: if no AS_PATH information is
  961. available an empty string, ie. "", is placed as value (instead of former
  962. "^$"). Similar stream-lining was done for communities. Many thanks to
  963. Brent Van Dussen and Elisa Jasinska for reporting the issue.
  964. ! fix, AMQP, MongoDB plugins: increased default refresh time to 60 secs,
  965. up from 10 and in line with SQL plugins value.
  966. ! fix, nfprobe plugin: IPv6 source/destination masks passed as IE #29 and
  967. #30 and not anymore as their IPv4 counterparts.
  968. ! fix, pmacct.c: clibuf variable now malloc'd at runtime so to not impact
  969. the data segment.
  970. ! fix, log.c: removed sbrk() calls when logging to Syslog.
  971. ! fix, pmacctd: If compiling against PF_RING, check and compile against
  972. libnuma and librt which are new requirement since version 5.6.2. Thanks
  973. to Joan Juvanteny for reporting the issue.
  974. ! fix, net_aggr.c: 'prev' array to keep track of hierarchies of networks
  975. was being re-initialized by some compilers. Thanks to Joan Juvanteny for
  976. reporting the issue.
  977. ! fix, MongoDB, JSON outputs: dst_host_country primitive was not properly
  978. shown. Patch is courtesy by Stig Thormodsrud.
  979. ! fix, pre_tag_map: a memory leak was found when reloading rules containing
  980. 'filter' keywords. Thanks to Matt Jenkins for his support resolving the
  981. issue.
  982. ! fix, server.c: countered a timing issue to ensure EOF is sent after data.
  983. Issue was originated by conjunction of non-blocking socket and multiple
  984. CPU cores. Thanks to Juan Camilo Cardona and Joel Ouellette Jr for their
  985. support.
  986. ! fix, acct.c: added length check to hash_crc32() of custom primitives as
  987. selective pmacct IMT client queries, ie. -M and -N, were failing to match
  988. entries. Thanks to Joel Ouellette Jr for his support.
  989. ! fix, nfacctd: NetFlow v9/IPFIX sampling correlation has been improved by
  990. placing system scoped sampling options in a separate table. Such table is
  991. queried if no matching sampler ID is found for a given <exporter IP addr,
  992. source ID>. Sampling-related fields (ie. sampler ID, interval, etc.) are
  993. now all supported if 1, 2 or 4 bytes long.
  994. ! fix, nfacctd: improved handling of the NAT64 case for NSEL. Thanks to
  995. Gregoire Leroy for his support.
  996. ! fix, nfacctd, sfacctd and BGP daemon: if IPv6 is enabled, IPv4 mapped is
  997. supported and can't obtain an IPv6 socket to listen to, retry with a IPv4
  998. one.
  999. 1.5.0rc2 -- 25-12-2013
  1000. + nfacctd: introduced support for variable-length IPFIX fields for custom-
  1001. defined aggregation primitives: 'string' semantics is supported and
  1002. maximum expected length of the field should be specified as 'len'
  1003. primitive definition. Also PENs are now supported: field_type can be
  1004. <value> or <PEN>:<value>. Finally, 'raw' semantics to print raw data,
  1005. fixed or variable length in hex format was added.
  1006. + pmacctd, uacctd: introducing custom-defined aggregation primitives in
  1007. libpcap and ULOG daemons. A new 'packet_ptr' keyword is supported in the
  1008. aggregate_primitives map for the task: it defines the base pointer in the
  1009. packet where to read the primitive value; intuitively, this is to be used
  1010. in conjunction with 'len'. The supported syntax is: <layer>:[<protocol
  1011. value>]+[<offset>]. 'layer' keys are: 'packet', 'mac', 'vlan', 'mpls',
  1012. 'l3', 'l4', 'payload'. Examples are provided in 'examples/primitives.lst'.
  1013. + nfacctd: introduced pro rating algorithm if sql_history is enabled and
  1014. nfacctd_time_new is disabled. Although ideal, the feature is disabled
  1015. by default for now and can be enabled by setting nfacctd_pro_rating to
  1016. true. Given a NetFlow/IPFIX flow duration greater than time-bins size as
  1017. configured by sql_history, bytes/packets counters are proportionally
  1018. distributed across all time-bins spanned by the flow. Many thanks to
  1019. Stefano Birmani for his support.
  1020. + Introducing index_maps: enables indexing of maps to increase lookup speeds
  1021. on large maps and/or sustained lookup rates. Indexes are automatically
  1022. defined basing on structure and content of the map, up to a maximum of 8.
  1023. Indexing of pre_tag_map, bgp_peer_src_as_map, flows_to_rd_map is supported.
  1024. + BGP daemon: introduced bgp_daemon_interval and bgp_daemon_batch config
  1025. directives: to prevent massive syncronization of BGP peers to contend
  1026. resources, BGP sessions are accepted in batches: these define the time
  1027. interval between any two batches and the amount of BGP peers in each batch
  1028. respectively.
  1029. + Introducing historical accounting offset (ie. sql_history_offset) to set
  1030. an offset to timeslots basetime. If history is set to 30 mins (by default
  1031. creating 10:00, 10:30, 11:00, etc. time-bins), with an offset of, say,
  1032. 900 seconds (so 15 mins) it will create 10:15, 10:45, 11:15, etc. time-
  1033. bins.
  1034. + print, MongoDB, SQL plugins: improved placement of tuples in the correct
  1035. table when historical accounting (ie. sql_history) and dynamic table
  1036. names (ie. sql_table) features are both in use.
  1037. + print, MongoDB, SQL plugins: dynamic file names (print plugin) and
  1038. tables (MongoDB and SQL plugins) can now include $peer_src_ip, $tag and
  1039. $tag2 variables: value is populated using the processed record value for
  1040. peer_src_ip, tag, tag2 primitives respectively.
  1041. + print plugin: introduced print_latest_file to point latest filename for
  1042. print_output_file time-series. Until 1.5.0rc1 selection was automagic.
  1043. But having introduced variable spool directory structures and primitives-
  1044. related variables the existing basic scheme of producing pointers had to
  1045. be phased-out.
  1046. + IMT plugin: added EOF in the client-server communication so to detect
  1047. uncompleted messages and print an error message. Thanks to Adam Jacob
  1048. Muller for his proposal.
  1049. + Introduced [nf|sf|pm]acctd_pipe size and bgp_daemon_pipe_size config
  1050. directives to define the size of the kernel socket used read traffic data
  1051. and for BGP messaging respectively.
  1052. + pmacctd, uacctd: mpls_top_label, mpls_bottom_label and mpls_stack_depth
  1053. primitives have been implemented.
  1054. + pmacctd, uacctd: GTP tunnel handler now supports inspection of GTPv1.
  1055. + pre_tag_map: results of evaluation of pre_tag_map, in case of a positive
  1056. match, overrides any tags passed by nfprobe/sfprobe plugins via NetFlow/
  1057. sFlow export.
  1058. + pre_tag_map: stack keyword now supports logical or operator (A | B) in
  1059. addition to sum (A + B).
  1060. + pre_tag_map: introduced 'mpls_pw_id' keyword to match the signalled MPLS
  1061. L2 VPNs Pseudowire ID. In NetFlow v9/IPFIX this is compared against IE
  1062. #249; in sFlow v5 this is compared against vll_vc_id field, extended MPLS
  1063. VC object.
  1064. + Introduced log notifications facility: allows to note down specific log
  1065. notifications have been sent so to prevent excessive repetitive output.
  1066. ! fix, plugin_hooks.c: plugin_buffer_size variables are bumped to u_int64_t
  1067. ! fix, plugin_hooks.c: improved protection of internal pmacct buffering
  1068. (plugin_buffer_size, plugin_pipe_size) from inconsistencies: buffer is now
  1069. also invalidated by the core process upon first writing into it. Thanks to
  1070. Chris Wilson for his support.
  1071. ! fix, plugin_hooks.c: a simple default value for plugin_pipe_size and
  1072. plugin_buffer_size is now picked if none is supplied. This is to get
  1073. around tricky estimates. 1.5.0rc1 release affected.
  1074. ! fix, ll.c: ntohl() done against a char pointer instead of u_int32_t one
  1075. in MPLS handler was causing incorrect parsing of labels. Thanks to Marco
  1076. Marzetti for his support.
  1077. ! fix, net_aggr.c: IPv6 networks debug messages now report correctly net
  1078. and mask information. Also IPv6 prefix to peer source/destination ASN was
  1079. crashing due to an incorrect pointer. Finally applying masks to IPv6
  1080. addresses was not done correctly. Thanks to Brent Van Dussen for
  1081. reporting the issue.
  1082. ! fix, classifiers: slightly optimized search_class_id_status_table() and
  1083. added warning message if the amount of classifiers exceeds configured
  1084. number of classifier_table_num (by default 256).
  1085. ! fix, pre_tag_map: if a JEQ can be resolved into multiple labels, stop to
  1086. the first occurrence.
  1087. ! fix, nfacctd, sfacctd: IPv6 was not being correctly reported due to a
  1088. re-definition of NF9_FTYPE_IPV6. 1.5.0rc1 release affected. Thanks to
  1089. Andrew Boey for reporting the issue.
  1090. ! fix, nfacctd: when historical accounting is enabled, ie. sql_history, not
  1091. assume anymore start and end timestamps to be of the same kind (ie. field
  1092. type #150/#151, #152/#153, etc.).
  1093. ! fix, BGP daemon: default BGP RouterID used if supplied bgp_daemon_ip is
  1094. "" or "::"
  1095. ! fix, BGP daemon: the socket opened to accept BGP peerings is restricted
  1096. to che core process (ie. closed upon instantiating the plugins). Thanks
  1097. to Olivier Benghozi for reporting the issue.
  1098. ! fix, BGP daemon: memory leak detected accepting vpnv4 and vpnv6 routes.
  1099. Thanks to Olivier Benghozi for his support solving the issue.
  1100. ! fix, BGP daemon: compiling the package without IPv6 support and sending
  1101. ipv6 AF was resulting in a buffer overrun. Thanks to Joel Krauska for his
  1102. support resolving the issue.
  1103. ! fix, IMT plugin: when gracefully exiting, ie. via a SIGINT signal, delete
  1104. the pipe file in place for communicating with the pmacct IMT client tool.
  1105. ! fix, print, MongoDB, AMQP plugins: saved_basetime variable initialized
  1106. to basetime value. This prevents P_eval_historical_acct() to consume much
  1107. resources during the first time-bin, if historical accounting is enabled
  1108. (ie. print_history). 1.5.0rc1 release affected.
  1109. ! fix, print, MongoDB and SQL plugins: purge function is escaped if there
  1110. are no elements on the queue to process.
  1111. ! fix, AMQP plugin: removed amqp_set_socket() call so to be able to compile
  1112. against rabbitmq-c >= 0.4.1
  1113. ! fix, MongoDB plugin: change of API between C driver version 0.8 and 0.7
  1114. affected mongo_create_index(). MongoDB C driver version test introduced.
  1115. Thanks to Maarten Bollen for reporting the issue.
  1116. ! fix, print plugin: SEGV was received if no print_output_file is specified
  1117. ie. print to standard output.
  1118. ! fix, MongoDB: optimized usage of BSON objects array structure.
  1119. ! fix, MongoDB plugin: brought a few numerical fields, ie. VLAN IDs, CoS,
  1120. ToS, etc. to integer representation, ie. bson_append_int(), from string
  1121. one, ie. bson_append_string(). Thanks to Job Snijders for his support.
  1122. ! fix, MySQL plugin: improved catching condition of sql_multi_value set too
  1123. little value. Thanks to Chris Wilson for reporting the issue.
  1124. ! fix, nfprobe plugin: catch ENETUNREACH errors instead of bailing out.
  1125. Patch is courtesy by Mike Jager.
  1126. 1.5.0rc1 -- 29-08-2013
  1127. + Introducing custom-defined aggregation primitives: primitives are defined
  1128. via a file pointed by aggregate_primitives config directive. The feature
  1129. applies to NetFlow v9/IPFIX fields only, and with a pre-defined length.
  1130. Semantics supported are: 'u_int' (unsigned integer, presented as decimal
  1131. number), 'hex' (unsigned integer, presented as hexa- decimal number), 'ip'
  1132. (IP address), 'mac' (MAC address)and 'str' (string). Syntax along with
  1133. examples are available in the 'examples/primitives.lst' file.
  1134. + Introducing JSON output in addition to tabular and CSV formats. Suitable
  1135. for injection in 3rd party tools, JSON has the advantage of being a self-
  1136. consisting format (ie. compared to CSV does not require a table title).
  1137. Library leveraged is Jansson, available at:
  1138. + Introducing RabbitMQ/AMQP pmacct plugin to publish network traffic data
  1139. to message exchanges. Unicast, broadcast, load-balancing scenarios being
  1140. supported. amqp_routing_key supports dynamic elements, like the value of
  1141. peer_src_ip and tag primitives or configured post_tag value, enabling
  1142. selective delivery of data to consumers. Messages are encoded in JSON
  1143. format.
  1144. + pre_tag_map (and other maps): 'ip' key, which is compared against the IP
  1145. address originating NetFlow/IPFIX or the AgentId field in sFlow, can now
  1146. be an IP prefix, ie. XXX.XXX.XXX.XXX/NN, so to apply tag statements to
  1147. set of exporters or to apply to any exporter. Many thanks to
  1148. Stefano Birmani for his support.
  1149. + Re-introducing support for Cisco ASA NSEL export. Previously it was just
  1150. a hack. Now most of the proper work done for Cisco NEL is being reused:
  1151. post_nat_src_host (field type #40001), post_nat_dst_host (field type
  1152. #40002), post_nat_src_port (field type #40003), post_nat_dst_port (field
  1153. type #40004), fw_event (variant of nat_event, field type #40005) and
  1154. timestamp_start (observation time in msecs, field type #323).
  1155. + Introducing MPLS-related aggregation primitives decoded from NetFlow v9/
  1156. IPFIX, mpls_label_top mpls_label_bottom and mpls_stack_depth, so to give
  1157. visibility in export scenarios on egress towards core, MPLS interfaces.
  1158. + mpls_vpn_rd: primitive value can now be sourced from NetFlow v9/IPFIX
  1159. field types #234 (ingressVRFID) and #235 (egressVRFID). This is in
  1160. addition to existing method to source value from a flow_to_rd_map file.
  1161. + networks_file: AS field can now be defined as "<peer_as>_<origin_as>",
  1162. Useful also to define (or override) elments of an internal port-to-port
  1163. traffic matrix.
  1164. + print plugin: creation of intermediate directory levels is now supported;
  1165. directories can contain dynamic time-based elements hence the amount of
  1166. variables in a given pathname was also lifted to 32 from 8.
  1167. + print plugin: introduced print_history configuration directive, which
  1168. supports same syntax as, for example, sql_history. When enabled, time-
  1169. related variables substitution of dynamic print_output_file names are
  1170. determined using this value instead of print_refresh_time one.
  1171. + Introducing IP prefix labels, ie. for custom grouping of own IP address
  1172. space. The feature can be enabled by a --enable-plabel when configuring
  1173. the package for compiling. Labels can be defined via a networks_file.
  1174. + mongo_user and mongo_passwd configuration directive have been added in
  1175. order to support authentication with MongoDB. If both are omitted, for
  1176. backward compatibility, authentication is disabled; if only one of the
  1177. two is specified instead, the other is set to its default value.
  1178. + Introducing mongo_indexes_file config directive to define indexes in
  1179. collections with dynamic name. If the collection does not exist yet, it
  1180. is created. Index names are picked by MongoDB.
  1181. + print plugin: introduced print_output_file_append config directive: if
  1182. set to true allows the plugin to append to an output file rather than
  1183. overwrite.
  1184. + bgp_agent_map: added bgp_port key to lookup a NetFlow agent also against
  1185. a BGP session port (in addition to BGP session IP address/router ID): it
  1186. aims to support scenarios where BGP sessions do NAT traverals.
  1187. + peer_dst_ip (BGP next-hop) can now be inferred by MPLS_TOP_LABEL_ADDR
  1188. (NetFlow v9/IPFIX field type #47). This field might replace BGP next-hop
  1189. when NetFlow is exported egress on MPLS-enabled core interfaces.
  1190. + Introducing [nf|pm|sf|u]acctd_proc_name config directives to define the
  1191. name of the core process (by default always set to 'default'). This is
  1192. the equivalent to instantiate named plugins but for the core process.
  1193. Thanks to Brian Rak for bringing this up.
  1194. + pre_tag_map: introduced key 'flowset_id' to tag NetFlow v9/IFPIX data
  1195. records basing on their flowset ID value, part of the flowset header.
  1196. + pmacct client: introduced '-V' command-line option to verify version,
  1197. build info and compile options passed to the configure script; also a
  1198. new -a option now allows to retrieve supported aggregation primitives
  1199. and their description.
  1200. + Check for mallopt() has been added at configure time. mallopt() calls
  1201. are introduced in order to disable glibc malloc() boundary checks.
  1202. ! flow_to_rd_map replaces iface_to_rd_map, increasing its scope: it is
  1203. now possible to map <MPLS bottom label, BGP next-hop> couples to BGP/
  1204. MPLS VPN Route Distinguishers (RD). This is in addition to existing
  1205. mapping method basing on <ingress router, input interface>.
  1206. ! fix, nfacctd, sfacctd: Setsocksize() call effectiveness is now verified
  1207. via a subsequent getsockopt(). If result is different than expected, an
  1208. informational log message is issued.
  1209. ! fix, building system: removed stale check for FreeBSD4 and introduced
  1210. check for BSD systems. If on a BSD system, -DBSD is now passed over to
  1211. the compiler.
  1212. ! fix, tee plugin: transparent mode now works on FreeBSD systems. Patch
  1213. is courtesy by Nikita V. Shirokov.
  1214. ! fix, peer_dst_ip: uninitialized pointer variable was causing unexpected
  1215. behaviours. Thanks to Maarten Bollen for his support resolving this.
  1216. ! fix, IMT plugin: selective queries with -M and -N switches verified not
  1217. working properly. Thanks to Acipia organization for providing a patch.
  1218. ! fix, sql_common.c: src_port and dst_port primitives correctly spelled if
  1219. used in conjunction with BGP primitives. Thanks to Brent Van Dussen and
  1220. Elisa Jasinska for flagging the issue.
  1221. ! fix, building system: added library checks in /usr/lib64 for OS's where
  1222. it is not linked to /lib where required.
  1223. ! fix, print, MongoDB and AMQP plugins: P_test_zero_elem() obsoleted.
  1224. Instead, the cache structure 'valid' field is used to commit entries to
  1225. the backend.
  1226. ! fix, nfacctd: in NetFlow v9/IPFIX, if no time reference is specified as
  1227. part of records, fall back to time reference in datagram header.
  1228. ! fix, MongoDB plugin: mongo_insert_batch() now bails out with MONGO_FAIL
  1229. if something went wrong while processing elements in the batch and an
  1230. error message is issued. Typical reason for such condition is batch is
  1231. too big for the resources, mainly memory, available. Thanks very much to
  1232. Maarten Bollen for his support.
  1233. ! fix, cfg_handlers.c: all functions parsing configuration directives, and
  1234. expecting string arguments, are now calling lower_string() so to act as
  1235. case insensitive.
  1236. ! fix, IPv6 & NetFlow exporter IP address: upon enabling IPv6, NetFlow
  1237. exporter IP addresses were written as IPv4-mapped IPv6 address. This was
  1238. causing confusion when composing maps since the 'ip' field would change
  1239. depending on whether IPv6 was enabled or not. This is now fixed and IPv4-
  1240. mapped IPv6 addresses are now internally translated to plain IPv4 ones.
  1241. ! fix, nfacctd: NetFlow v9/IPFIX source/destination peer ASN information
  1242. elements have been found mixed up and are now in proper order.
  1243. 0.14.3 -- 03-05-2013
  1244. + tee plugin: a new tee_receivers configuration directive allows multiple
  1245. receivers to be defined. Receivers can be optionally grouped, for example
  1246. for load-balancing (rr, hash) purposes, and attached a list of filters
  1247. (via tagging). The list is fully reloadable at runtime.
  1248. + A new pkt_len_distrib aggregation primitive is introduced: it works by
  1249. defining length distribution bins, ie. "0-999,1000-1499,1500-9000" via
  1250. the new pkt_len_distrib_bins configuration directive. Maximum amount
  1251. of bins that can be defined is 255; lengths must be within the range
  1252. 0-9000.
  1253. + Introduced NAT primitives to support Cisco NetFlow Event Logging (NEL),
  1254. for Carrier Grade NAT (CGNAT) scenarios: nat_event, post_nat_src_host,
  1255. post_nat_dst_host, post_nat_src_port and post_nat_dst_port. Thanks to
  1256. Simon Lockhart for his input and support developing the feature.
  1257. + Introduced timestamp primitives (to msec resolution) to support generic
  1258. logging functions: timestamp_start, timestamp_end (timestamp_end being
  1259. currently applicable only to traffic flows). These primitives must not
  1260. be confused with existing sql_history timestamps which are meant for the
  1261. opposite function instead, temporal aggregation.
  1262. + networks_file: introduced support for (BGP) next-hop (peer_dst_ip) in
  1263. addition to existing fields. Improved debug output. Also introduced a
  1264. new networks_file_filter feature to make networks_file work as a filter
  1265. in addition to its resolver functionality: if set to true net and host
  1266. values not belonging to defined networks are zeroed out. See UPGRADE
  1267. document for backward compatibility.
  1268. + BGP daemon: added support for IPv6 NLRI and IPv6 BGP next-hop elements
  1269. for rfc4364 BGP/MPLS Virtual Private Networks.
  1270. + MongoDB plugin: introduced mongo_insert_batch directive to define the
  1271. amount of elements to be inserted per batch - allowing the plugin to
  1272. scale better. Thanks for the strong support to Michiel Muhlenbaumer and
  1273. Job Snijders.
  1274. + pre_tag_map: 'set_qos' feature introduced: matching network traffic is
  1275. set 'tos' primitive to the specified value. This is useful if collecting
  1276. ingress NetFlow/IPFIX at both trusted and untrusted borders, allowing to
  1277. selectively override ToS values at untrusted ones. For consistency,
  1278. pre_tag_map keys id and id2 have been renamed to set_tag and set_tag2;
  1279. legacy jargon is still supported for backward compatibility.
  1280. + sfacctd: improved support for L2 accounting, ethernet length is being
  1281. committed as packet length; this information gets replaced by any length
  1282. information will come from upper layers, if any is reported. Thanks to
  1283. Daniel Swarbrick for his support.
  1284. + nfacctd: introduced nfacctd_peer_as directive to value peer_src_as and
  1285. peer_dst_as primitives from NetFlow/IPFIX export src_as and dst_as
  1286. values respectively (ie. as a result of a "ip flow-export .. peer-as"
  1287. config on the exporter). The directive can be plugin-specific.
  1288. + print, memory plugins: print_output_separator allows to select separator
  1289. for CSV outputs. Default comma separator is generally fine except for
  1290. BGP AS-SET representation.
  1291. ! Building sub-system: two popular configure switches, --enable-threads
  1292. and --enable-64bit, are now set to true by default.
  1293. ! fix, print & mongodb plugins: added missing cases for src_net and dst_net
  1294. primitives. Thanks to John Hess for his support.
  1295. ! fix, SQL plugins: improved handling of fork() calls when return value
  1296. is -1 (fork failed). Many thanks to Stefano Birmani for his valuable
  1297. support troubleshooting the issue.
  1298. ! fix, ISIS daemon: linked list functions got isis_ prefix in order to
  1299. prevent namespace clashes with other libraries (ie. MySQL) we link
  1300. against. Thanks to Stefano Birmani for reporting the issue.
  1301. ! fix, tee plugin: can't bridge AFs when in transparent mode is not fatal
  1302. error condition anymore to tackle transient interface conditions. Error
  1303. message is throttled to once per 60 secs. Thanks to Evgeniy Kozhuhovskiy
  1304. for his support troubleshooting the issue.
  1305. ! fix, nfacctd: extra length checks introduced when parsing NetFlow v9/
  1306. IPFIX options and data template flowsets. Occasional daemon crashes were
  1307. verified upon receipt of malformed/incomplete template data.
  1308. ! fix: plugins now bail out with an error message if core process is found
  1309. dead via a getppid() check.
  1310. - nfacctd_sql_log feature removed. The same can now be achieved with the
  1311. use of proper timestamp primitives (see above).
  1312. 0.14.2 -- 14-01-2013
  1313. + pmacct opens to MongoDB, a leading noSQL document-oriented database
  1314. via a new 'mongodb' plugin. Feature parity is maintained with all
  1315. existing plugins. The QUICKSTART doc includes a brief section on how
  1316. to getting started with it. Using MongoDB >= 2.2.0 is recommended;
  1317. MongoDB C driver is required.
  1318. + GeoIP lookups support has been introduced: geoip_ipv4 and geoip_ipv6
  1319. config directives now allow to load Maxmind IPv4/IPv6 GeoIP database
  1320. files; two new traffic aggregation primitives are added to support the
  1321. feature: src_host_country and dst_host_country. Feature implemented
  1322. against all deamons and all plugins and supports both IPv4 and IPv6.
  1323. Thanks to Vincent Bernat for his patches and precious support.
  1324. + networks_file: user-supplied files to define IP networks and their
  1325. associations to ASNs (optional) has been hooked up to the 'fallback'
  1326. (longest match wins) setting of [pm|u|sf|nf]acctd_net, [pm|u]acctd_as
  1327. and [sf|nf]acctd_as_new. Thanks to John Hess for his support.
  1328. + A new sampling_rate traffic aggregation primitive has been introduced:
  1329. to report on the sampling rate to be applied to renormalize counters
  1330. (ie. useful to support troubleshooting of untrusted node exports and
  1331. hybrid scenarios where a partial sampling_map is supplied). If renorm
  1332. of counters is enabled (ie. [n|s]facctd_renormalize set to true) then
  1333. sampling_rate will show as 1 (ie. already renormalized).
  1334. + sql_table, print_output_file, mongo_table: dynamic table names are
  1335. now enriched by a $ref variable, populated with the configured value
  1336. for refresh time, and a $hst variable, populated with the configured
  1337. value for sql_history (in secs).
  1338. + Solved the limit of 64 traffic aggregation primitives: the original
  1339. 64 bits bitmap is now split in a 16 bits index + 48 bits registry
  1340. with multiple entries (currently 2). cfg_set_aggregate() and, in
  1341. future, cfg_get_aggregate() functions are meant to safely manipulate
  1342. the new bitmap structure and detect mistakes in primitives definition.
  1343. ! fix, print plugin: removed print_output_file limitation to 64 chars.
  1344. Now maximum filename length is imposed by underlying OS.
  1345. ! fix, print plugin: primitives are selectively enabled for printing
  1346. based on 'aggregate' directive.
  1347. ! fix, print plugin: pointer to latest file been generated is updated
  1348. at very last in the workflow.
  1349. ! fix, ip_flow.c: incorrect initialization for IPv6 flow buffer. Thanks
  1350. to Mike Jager for reporting the issue and providing a patch.
  1351. ! fix, pre_tag_map: improved matching of pre_tag_map primitives against
  1352. IPFIX fields. Thanks to Nikita V Shirokov for reporting the issue.
  1353. ! fix, nfprobe plugin: improved handling of unsuccessful send() calls
  1354. in order to prevent file descriptors depletion and log failure cause.
  1355. Patch is courtesy by Mike Jager.
  1356. ! fix, nfacctd: gracefully handling the case of NetFlow v9/IPFIX flowset
  1357. length of zero; unproper handling of the condition was causing nfacctd
  1358. to infinite loop over the packet; patch is courtesy by Mike Jager.
  1359. ! fix, Setsocksize(): setsockopt() replaces Setsocksize() in certain
  1360. cases and Setsocksize() fix to len parameter. Patch is courtesy by
  1361. Vincent Bernat
  1362. 0.14.1 -- 03-08-2012
  1363. + nfacctd: introduced support for IPFIX variable-length IEs (RFC5101),
  1364. improved support for IPFIX PEN IEs.
  1365. + nfacctd, sfacctd: positive/negative caching for bgp_agent_map and
  1366. sampling_map is being introduced. Cache entries are invalidated upon
  1367. reload of the maps.
  1368. + bgp_agent_map: resolution of IPv4 NetFlow agents to BGP speakers
  1369. with IPv6 sessions is now possible. This is to support dual-stack
  1370. network deployments. Also the keyword 'filter' is introduced and
  1371. supported values are only 'ip' and 'ip6'.
  1372. + nfacctd: etype primitive can be populated from IP_PROTOCOL_VERSION,
  1373. ie. Field Type #60, in addition to ETHERTYPE, ie. Field Type #256.
  1374. Should both be present the latter has priority over the former.
  1375. + print plugin: introduced a pointer to the latest filename in the set,
  1376. ie. in cases when variable filenames are specified. The pointer comes
  1377. in the shape of a symlink called "<plugin name>-latest".
  1378. ! fix, pretag_handlers.c: BGP next-hop handlers are now hooked to the
  1379. longest-match mechanism for destination IP prefix.
  1380. ! fix, net_aggr.c: defining a networks_file configuration directive in
  1381. conjunction with --enable-ipv6 was causing a SEGVs. This is now solved.
  1382. ! fix, uacctd: cache routine is now being called in order to resolve
  1383. in/out interface ifindexes. Patch is courtesy by Stig Thormodsrud.
  1384. ! fix, BGP daemon: bgp_neighbors_file now lists also IPv6 BGP peerings.
  1385. ! fix, sql_common.c: SQL writers due to safe action are now logged with
  1386. a warning message rather than debug.
  1387. ! fix, PostgreSQL table schemas: under certain conditions, default
  1388. definition of stamp_inserted was generating a 'date/time field value
  1389. out of range: "0000-01-01 00:00:00"' error. Many thanks to Marcello
  1390. di Leonardo for reporting the issue and providing a fix.
  1391. ! fix, IS-IS daemon: sockunion_print() function was found not portable
  1392. and has been removed.
  1393. ! fix, BGP daemon: memcpy() replaced by ip6_addr_cpy() upon writing to
  1394. sockaddr_in6 structures.
  1395. ! fix, EXAMPLES document has been renamed QUICKSTART for disambiguation
  1396. on filesystems where case-sensitive names are not supported.
  1397. ! Several code cleanups. Patches are courtesy by Osama Abu Elsorour
  1398. and Ryan Steinmetz.
  1399. 0.14.0 -- 11-04-2012
  1400. + pmacct now integrates an IS-IS daemon within collectors; the daemon
  1401. is being run as a parallel thread within the collector core process;
  1402. a single L2 P2P neighborship, ie. over a GRE tunnel, is supported;
  1403. it implements P2P Hello, CSNP and PSNP - and does not send any LSP
  1404. information out. The daemon is currently used for route resolution.
  1405. It is well suited to several case-studies, popular one being: more
  1406. specific internal routes are carried within the IGP while they are
  1407. summarized in BGP crossing cluster boundaries.
  1408. + A new aggregation primitive 'etype' has been introduced in order to
  1409. support accounting against the EtherType field of Ethernet frames.
  1410. The implementation is consistent across all data collection methods
  1411. and backends.
  1412. + sfacctd: introduced support for samples generated on ACL matches in
  1413. Brocade (sFlow sample type: Enterprise: #1991, Format: #1). Thanks
  1414. to Elisa Jasinska and Brent Van Dussen for their support.
  1415. + sfacctd, pre_tag_map: introduced sample_type key. In sFlow v2/v4/v5
  1416. this is compared against the sample type field. Value is expected
  1417. in <Enterprise>:<Format> notation.
  1418. ! fix, signals.c: ignoring SIGINT and SIGTERM in my_sigint_handler()
  1419. to prevent multiple calls to fill_pipe_buffer(), condition that can
  1420. cause pipe buffer overruns. Patch is courtesy by Osama Abu Elsorour.
  1421. ! fix, pmacctd: tunnel registry now correctly supports multiple tunnel
  1422. definitions for the same stack level.
  1423. ! fix, print plugin: cos field now correctly shows up in the format
  1424. title while CSV format is selected and L2 primitives are enabled.
  1425. ! fix, util.c: a feof() check has been added to the fread() call in
  1426. read_SQLquery_from_file(); thanks to Elisa Jasinska and Brent Van
  1427. Dussen for their support.
  1428. ! fix, nfprobe: NetFlow output socket is now re-opened after failing
  1429. send() calls. Thanks to Maurizio Molina for reporting the problem.
  1430. ! fix, sfacctd: length checks have been imporved while extracting
  1431. string tokens (ie. AS-PATH and BGP communities) from sFlow Extended
  1432. Gateway object. Thanks to Duncan Small for his support.
  1433. 0.14.0rc3 -- 07-12-2011
  1434. + BGP daemon: BGP/MPLS VPNs (rfc4364) implemented! This encompasses both
  1435. RIB storage (ie. virtualization layer) and lookup. bgp_iface_to_rd_map
  1436. map correlates <router IP, input/output interfaces (ifIndex)> couples
  1437. to Route Distinguishers (RDs). RD encapsulation types #0 (2-bytes ASN),
  1438. #1 (IP address) and #2 (4-bytes ASN) are supported. Examples provided:
  1439. examples/ and EXAMPLES files.
  1440. + mpls_vpn_rd aggregation primitive has been added to the set. Also this
  1441. is being supported key in Pre-Tagging (pre_tag_map).
  1442. + print plugin: introduced print_output_file feature to write statistics
  1443. to files. Output is text, formatted or CSV. Filenames can contain time-
  1444. based variables to make them dynamic. If filename is static instead,
  1445. content is overwritten over time.
  1446. + print plugin: introduced print_time_roundoff feature to align time slots
  1447. nicely, same as per the sql_history_roundoff directive.
  1448. + print plugin: introduced print_trigger_exec feature to execute custom
  1449. scripts at each print_refresh_time interval (ie. to process, expire,
  1450. gzip, etc. files). Feature is in sync with wrap-up of data commit to
  1451. screen or files.
  1452. + pmacctd: introduced support for DLT_LOOP link-type (ie. OpenBSD tunnel
  1453. interfaces). Thanks to Neil Reilly for his support.
  1454. + uacctd: a cache of ifIndex is introduced. Hash structure with conflict
  1455. chains and short expiration time (ie. to avoid getting tricked by cooked
  1456. interfaces devices a-la ppp0). The cache is an effort to gain speed-ups.
  1457. Implementation is courtesy by Stephen Hemminger, Vyatta.
  1458. + Logging: introduced syslog-like timestamping when writing directly to
  1459. files. Also a separate FD per process is used and SIGHUP elicits files
  1460. reopening: all aimed at letting proper logs rotation by external tools.
  1461. + Introduced plugin_pipe_backlog configuration directive: it induces a
  1462. backlog of buffers on the pipe before actually releasing them to the
  1463. plugin. The strategy helps optimizing inter-process communications, ie.
  1464. when plugins are quicker processing data than the Core process.
  1465. ! fix, peer_src_ip primitive: has been disconnected from [ns]facctd_as_new
  1466. mechanism in order to ensure it's always representing a reference to the
  1467. NetFlow or sFlow emitter.
  1468. ! fix, nfprobe: input and output VLAN ID field types have been aligned to
  1469. RFC3954, which appears to be also retroactively supported by IPFIX. The
  1470. new field types are #58 and #59 respectively. Thanks to Maurizio Molina
  1471. for pointing the issue out.
  1472. ! fix, IMT plugin: fragmentation of the class table over multiple packets
  1473. to the pmacct IMT client was failing and has been resolved.
  1474. ! fix, nfprobe: individual flows start and end timestamps are now filled
  1475. to the msec resolution. Thanks to Daniel Aschwanden for having reported
  1476. the issue.
  1477. ! fix, uacctd: NETLINK_NO_ENOBUFS is set to prevent the daemon being
  1478. reported about ENOBUFS events by the underlying operating system. Works
  1479. on kernels 2.6.30+. Patch is courtesy by Stephen Hemminger, Vyatta.
  1480. ! fix, uacctd: get_ifindex() can now return values greater than 2^15. Patch
  1481. is courtesy by Stephen Hemminger, Vyatta.
  1482. ! fix, pmacctd, uacctd: case of zero IPv6 payload in conjunction with no
  1483. IPv6 next header is now supported. Thanks to Quirin Scheitle for having
  1484. reported the issue.
  1485. - Support for is_symmetric aggregation primitive is discontinued.
  1486. 0.14.0rc2 -- 26-08-2011
  1487. + sampling_map feature is introduced, allowing definition of static traffic
  1488. sampling mappings. Content of the map is reloadable at runtime. If a
  1489. specific router is not defined in the map, the sampling rate advertised
  1490. by the router itself, if any, is applied.
  1491. + nfacctd: introduced support for 16 bits SAMPLER_IDs in NetFlow v9/IPFIX;
  1492. this appears to be the standard length with IOS-XR.
  1493. + nfacctd: introduced support for (FLOW)_SAMPLING_INTERVAL fields as part
  1494. of the NetFlow v9/IPFIX data record. This case is not prevented by the
  1495. RFC although such information is typically exported as part of options.
  1496. It appears some probes, ie. FlowMon by Invea-Tech, are getting down this
  1497. way.
  1498. + nfacctd, sfacctd: nfacctd_as_new and sfacctd_as_new got a new 'fallback'
  1499. option; when specified, lookup of BGP-related primitives is done against
  1500. BGP first and, if not successful, against the export protocol.
  1501. + nfacctd, sfacctd: nfacctd_net and sfacctd_net got a new 'fallback' option
  1502. that when specified looks up network-related primitives (prefixes, masks)
  1503. against BGP first and, if not successful, against the export protocol. It
  1504. gets useful for resolving prefixes advertised only in the IGP.
  1505. + sql_num_hosts feature is being introduced: defines, in MySQL and SQLite
  1506. plugins, whether IP addresses should be left numerical (in network bytes
  1507. ordering) or converted into strings. For backward compatibility, default
  1508. is to convert them into strings.
  1509. + print_num_protos and sql_num_protos configuration directives have been
  1510. introduced to allow to handle IP protocols (ie. tcp, udp) in numerical
  1511. format. The default, backward compatible, is to look protocol names up.
  1512. The feature is built against all plugins and can also be activated via
  1513. the '-u' commandline switch.
  1514. ! fix, nfacctd: NetFlow v9/IPFIX sampling option parsing now doesn't rely
  1515. anymore solely on finding a SamplerID field; as an alternative, presence
  1516. of a sampling interval field is also checked. Also a workaround is being
  1517. introduced for sampled NetFlow v9 & C7600: if samplerID within a data
  1518. record is defined and set to zero and no match was possible, then the
  1519. last samplerID defined is returned.
  1520. ! nfacctd: (FLOW)_SAMPLING_INTERVAL fields as part of the NetFlow v9/IPFIX
  1521. data record are now supported also 16-bits long (in addition to 32-bits).
  1522. ! fix, SQL plugins: sql_create_table() timestamp has been aligned with SQL
  1523. queries (insert, update, lock); furthermore sql_create_table() is invoked
  1524. every sql_refresh_time instead of every sql_history. Docs updated. Thanks
  1525. to Luis Galan for having reported the issue.
  1526. ! fix, pmacct client: error code when connection is refused on UNIX socket
  1527. was 0; it has been changed to 1 to reflect the error condition. Thanks
  1528. to Mateusz Viste for reporting the issue.
  1529. ! fix, building system: CFLAGS were not always honoured. Patch is courtesy
  1530. of Etienne Champetier
  1531. ! fix, ll.c: empty return value was causing compiler with certain flags to
  1532. complain about the issue. Patch is courtesy of Ryan Steinmetz.
  1533. 0.14.0rc1 -- 31-03-2011
  1534. + IPFIX (IETF IP Flow Information Export protocol) replication and
  1535. collector capabilities have been introduced as part of nfacctd, the
  1536. NetFlow accounting daemon of the pmacct package.
  1537. + nfprobe plugin: initial IPFIX export implementation. This is called
  1538. via a 'nfprobe_version: 10' configuration directive. pmacctd, the
  1539. promiscuous mode accounting daemon, and uacctd, the ULOG accounting
  1540. daemon, both part of the pmacct package are now supported.
  1541. + Oracle's BrekeleyDB 11gR2 offers a perfect combination of technologies
  1542. by including an SQL API that is fully compatible with SQLite. As a
  1543. result pmacct now opens to BerkeleyDB 5.x via its SQLite3 plugin.
  1544. + sfacctd: BGP-related traffic primitives (AS Path, local preference,
  1545. communities, etc.) are now read from sFlow Extended Gateway object if
  1546. sfacctd_as_new is set to false (default).
  1547. + nfacctd, sfacctd: source and destination peer ASNs are now read from
  1548. NetFlow or sFlow data if [ns]facctd_as_new is set to false (default).
  1549. + nfacctd: introduced support for NetFlow v9/IPFIX source and destination
  1550. peer ASN field types 128 and 129. The support is enabled at runtime by
  1551. setting to 'false' (default) the 'nfacctd_as_new' directive.
  1552. + sfacctd: f_agent now points sFlow Agent ID instead of source IP address;
  1553. among the other things, this allows to compare BGP source IP address/BGP
  1554. Router-ID against the sFlow Agent ID.
  1555. + PostgreSQL plugin: 'sql_delimiter' config directive being introduced:
  1556. if sql_use_copy is true, uses the supplied character as delimiter.Useful
  1557. in cases where the default delimiter is part of any of the supplied
  1558. strings.
  1559. + pmacct client: introduced support for Comma-Separated Values (CSV) output
  1560. in addition to formatted-text. A -O commandline switch allows to enable
  1561. the feature.
  1562. ! fix, MySQL/PostgreSQL/SQLite3 plugins: insert of data into the database
  1563. can get arbitrarily delayed under low traffic conditions. Many Thanks
  1564. to Elisa Jasinska and Brent Van Dussen for their great support in solving
  1565. the issue.
  1566. ! fix, BGP daemon: multiple BGP capabilities per capability announcement
  1567. were not supported - breaking compliancy with RFC5492. The issue was
  1568. only verified against a OpenBGPd speaker. Patch is courtesy of Manuel
  1569. Guesdon.
  1570. ! fix, initial effort made to document uacctd, the ULOG accounting daemon
  1571. 0.12.5 -- 28-12-2010
  1572. + nfacctd: introduced support for NAT L3/L4 field values via xlate_src
  1573. and xlate_dst configuration directives. Implementation follows IPFIX
  1574. standard for IPv4 and IPv6 (field types 225, 226, 227, 228, 281 and
  1575. 282).
  1576. + nfacctd: Cisco ASA NetFlow v9 NSEL field types 40001, 40002, 40003,
  1577. 40004 and IPFIX/Cisco ASA NetFlow v9 NSEL msecs absolute timestamps
  1578. field types 152, 153 and 323 have been added.
  1579. + nfacctd: introduced support for 'new' TCP/UDP source/destination ports
  1580. (field types 180, 181, 182, 183), as per IPFIX standard, basing on the
  1581. L4 protocol value (if any is specified as part of the export; otherwise
  1582. assume L4 is not TCP/UDP).
  1583. + nfacctd, nfprobe: introduced support for application classification
  1584. via NetFlow v9 field type #95 (application ID) and application name
  1585. table option. This feature aligns with Cisco NBAR-NetFlow v9
  1586. integration feature.
  1587. + nfacctd: introduced support for egress bytes and packet counters (field
  1588. types 23, 24) basing on the direction value (if any is specified as
  1589. part of the export; otherwise assume ingress as per RFC3954).
  1590. + nfprobe: egress IPv4/IPv6 NetFlow v9 templates have been introduced;
  1591. compatibility with Cisco (no use of OUT_BYTES, OUT_OUT_PACKETS) taken
  1592. into account.
  1593. + nfacctd: added support for egress datalink NetFlow v9 fields basing
  1594. on direction field.
  1595. + nfacctd, sfacctd: aggregate_filter can now filter against TCP flags;
  1596. also, [ns]facctd_net directive can now be specified per-plugin.
  1597. + BGP daemon: introduced support for IPv6 transport of BGP messaging.
  1598. + BGP daemon: BGP peer information is now linked into the status table
  1599. for caching purposes. This optimization results in good CPU savings
  1600. in bigger deployments.
  1601. ! fix, nfacctd, sfacctd: daemons were crashing on OpenBSD platform upon
  1602. setting an aggregate_filter configuration directive. Patch is courtesy
  1603. of Manuel Pata.
  1604. ! fix, xflow_status.c: status entries were not properly linked to the
  1605. hash conflict chain resulting in a memory leak. However the maximum
  1606. number of table entries set by default was preventing the structure
  1607. to grow undefinitely.
  1608. ! fix, sql_common.c: increased buffer size available for sql_table_schema
  1609. from 1KB to 8KB. Thanks to Michiel Muhlenbaumer his support.
  1610. ! fix, bgp_agent_map has been improved to allow mapping of NetFlow/sFlow
  1611. agents making use of IPv6 transport to either a) IPv4 transport address
  1612. of BGP sessions or b) 32-bit BGP Router IDs. Mapping to IPv6 addresses
  1613. is however not (yet) possible.
  1614. ! fix, nfprobe: encoding of NetFlow v9 option scope has been improved;
  1615. nfprobe source IPv4/IPv6 address, if specified via nfprobe_source_ip
  1616. directive, is now being written.
  1617. ! fix, util.c: string copies in trim_spaces(), trim_all_spaces() and
  1618. strip_quotes() have been rewritten more safely. Patch is courtesy of
  1619. Dmitry Koplovich.
  1620. ! fix, sfacctd: interface format is now merged back into interface value
  1621. fields so to ease keeping track of discards (and discard reasons) and
  1622. multicast fanout.
  1623. ! fix, MySQL, SQLite3 plugins: sql table version 8 issued to provide
  1624. common naming convention when mapping primitives to database fields
  1625. among the supported RDBMS base. Thanks to Chris Wilson for his support.
  1626. ! fix, pmacct client: numeric variables output converted to unsigned
  1627. from signed.
  1628. ! fix, nfacctd_net, sfacctd_net: default value changed from null (and
  1629. related error message) to 'netflow' for nfacctd_net and 'sflow' for
  1630. sfacctd_net.
  1631. ! fix, nfacctd, sfacctd: aggregate_filter was not catching L2 primitives
  1632. (VLAN, MAC addresses) when performing egress measurements.
  1633. 0.12.4 -- 01-10-2010
  1634. + BGP daemon: a new memory model is introduced by which IP prefixes
  1635. are being shared among the BGP peers RIBs - leading to consistent
  1636. memory savings whenever multiple BGP peers export full tables due
  1637. to the almost total overlap of information. Longest match nature
  1638. of IP lookups required to raise BGP peer awareness of the lookup
  1639. algorithm. Updated INTERNALS document to support estimation of the
  1640. memory footprint of the daemon.
  1641. + BGP daemon: a new bgp_table_peer_buckets configuration directive
  1642. is introduced: per-peer routing information is attached to IP
  1643. prefixes and now hashed onto buckets with conflict chains. This
  1644. parameter sets the number of buckets of such hash structure; the
  1645. value is directly related to the number of expected BGP peers,
  1646. should never exceed such amount and is best set to 1/10 of the
  1647. expected number of peers.
  1648. + nfprobe: support has been added to export direction field (NetFlow
  1649. v9 field type #61); its value, 0=ingress 1=egress, is determined
  1650. via nfprobe_direction configuration directive.
  1651. + nfacctd: introduced support for Cisco ASA bytes counter, NetFlow v9
  1652. field type #85. Thanks to Ralf Reinartz for his support.
  1653. + nfacctd: improved flow recognition heuristics for cases in which
  1654. IPv4/IPv6/input/output data are combined within the same NetFlow
  1655. v9 template. Thanks to Carsten Schoene for his support.
  1656. ! fix, BGP daemon: bgp_nexthop_followup was not working correctly if
  1657. pointed to a non-existing next-hop.
  1658. ! fix, nfv9_template.c: ignoring unsupported NetFlow v9 field types;
  1659. improved template logging. Thanks to Ralf Reinartz for his support.
  1660. ! fix, print plugin: support for interfaces and network masks has
  1661. been added. Numeric variables output converted to unsigned from
  1662. signed.
  1663. 0.12.3 -- 28-07-2010
  1664. + 'cos' aggregation primitive has been implemented providing support
  1665. for 802.1p priority. Collection is supported via sFlow, libpcap and
  1666. ULOG; export is supported via sFlow.
  1667. + BGP daemon: TCP MD5 signature implemented. New 'bgp_daemon_md5_file'
  1668. configuration directive is being added for the purpose of defining
  1669. peers and their respective MD5 keys, one per line, in CSV format.
  1670. The map is reloadable at runtime: existing MD5 keys are removed via
  1671. setsockopt(), new ones are installed as per the newly supplied map.
  1672. Sample map added in 'examples/bgp_md5.lst.example'.
  1673. + BGP daemon: added support for RFC3107 (SAFI=4 label information) to
  1674. enable receipt of labeled IPv4/IPv6 unicast prefixes.
  1675. + nfprobe, sfprobe: introduced the concept of traffic direction. As a
  1676. result, [ns]fprobe_direction and [ns]fprobe_ifindex configuration
  1677. directives have been implemented.
  1678. + [ns]fprobe_direction defines traffic direction. It can be statically
  1679. defined via 'in' or 'out' keywords; values can also be dynamically
  1680. determined through a pre_tag_map (1=input, 2=output) by means of
  1681. 'tag' and 'tag2' keywords.
  1682. + [ns]fprobe_ifindex either statically associate an interface index
  1683. (ifIndex) to a given [ns]fprobe plugin or semi-dynamically via
  1684. lookups against a pre_tag_map by means of 'tag' and 'tag2' keywords.
  1685. + sfprobe: sfprobe_ifspeed configuration directive is introduced and
  1686. aimed at statically associating an interface speed to an sfprobe
  1687. plugin.
  1688. + sfprobe: Switch Extension Header support added. Enabler for this
  1689. development was support for 'cos' and in/out direction. Whereas
  1690. VLAN information was already supported as an aggregation primitive.
  1691. + sfprobe: added support for Counter Samples for multiple interfaces.
  1692. Sampling function has been brought to the plugin so that Counter
  1693. Samples can be populated with real bytes/packets traffic levels.
  1694. ! nfprobe, sfprobe: send buffer size is now aligned to plugin_pipe_size,
  1695. if specified, providing a way to tune buffers in case of sustained
  1696. exports.
  1697. ! fix, addr.c: pm_ntohll() and pm_htonll() routines rewritten. These
  1698. are aimed at changing byte ordering of 64-bit variables.
  1699. ! fix, BGP daemon: support for IPv6 global address/link-local address
  1700. next-hops as part of MP_REACH_NLRI parsing.
  1701. ! fix, cfg_handlers.c: bgp_daemon and bgp_daemon_msglog parsing was
  1702. not correct, ie. enabled if specified as 'false'. Thanks to Brent
  1703. Van Dussen for reporting the issue.
  1704. ! fix, bgp.c: found a CPU hog issue caused by missing cleanup of the
  1705. select() descriptors vector.
  1706. ! fix, pmacct.c: in_iface/out_iface did erroneously fall inside a
  1707. section protected by the "--disable-l2" switch. Thanks to Brent
  1708. Van Dussen for reporting the issue.
  1709. 0.12.2 -- 27-05-2010
  1710. + A new 'tee' plugin is introduced bringing both NetFlow and sFlow
  1711. replication capabilities to pmacct. It supports transparent mode
  1712. (tee_transparent), coarse-grained filtering capabilities via the
  1713. Pre-Tagging infrastructure. Quickstart guide is included as part
  1714. of the EXAMPLES file (chapter XII).
  1715. + nfprobe, sfprobe: introduced support for export of the BGP next-hop
  1716. information. Source data selection for BGP next-hop is being linked
  1717. to [pmacctd_as|uacctd_as] configuration directive. Hence it must be
  1718. set to 'bgp' in order for this feature to work.
  1719. + nfprobe, sfprobe, BGP daemon: new set of features (nfprobe_ipprec,
  1720. sfprobe_ipprec, bgp_daemon_ipprec) allows to mark self-originated
  1721. sFlow, NetFlow and BGP datagrams with the supplied IP precedence
  1722. value.
  1723. + peer_src_ip (IP address of the NetFlow emitter, agent ID of the
  1724. sFlow emitter) and peer_dst_ip (BGP next-hop) can now be filled
  1725. from NetFlow/sFlow protocols data other than BGP. To activate the
  1726. feature nfacctd_as_new/sfacctd_as_new have to be 'false' (default
  1727. value), 'true' or 'file'.
  1728. + print plugin: introduced support for Comma-Separated Values (CSV)
  1729. output in addition to formatted-text. A new print_output feature
  1730. allows to switch between the two.
  1731. + pmacctd: improved 802.1ad support. While recursing, outer VLAN is
  1732. always reported as value of the 'vlan' primitive.
  1733. ! fix, pmacctd: 802.1p was kept integral part of the 'vlan' value.
  1734. Now a 0x0FFF mask is applied in order to return only the VLAN ID.
  1735. ! fix, pkt_handlers.c: added trailing '\0' symbol when truncating
  1736. AS-PATH and BGP community strings due to length constraints.
  1737. ! fix, sql_common.c: maximum SQL writers warning message was never
  1738. reached unless a recovery method is specifited. Thanks to Sergio
  1739. Charpinel Jr for reporting the issue.
  1740. ! fix, MySQL and PostgreSQL plugins: PGRES_TUPLES_OK (PostgreSQL)
  1741. and errno 1050 (MySQL) are now considered valid return codes when
  1742. dynamic tables are involved (ie. sql_table_schema). Thanks to
  1743. Sergio Charpinel Jr for his support.
  1744. ! fix, BGP daemon: pkt_bgp_primitives struct has been explicitely
  1745. 64-bit aligned. Mis-alignment was causing crashes when buffering
  1746. was enabled (plugin_buffer_size). Verified on Solaris/sparc.
  1747. 0.12.1 -- 07-04-2010
  1748. + Input/output interfaces (SNMP indexes) have now been implemented
  1749. natively; it's therefore not required anymore to pass through the
  1750. (Pre-)tag infrastructure. As a result two aggregation primitives
  1751. are being introduced: 'in_iface' and 'out_iface'.
  1752. + Support for source/destination IP prefix masks is introduced via
  1753. two new aggregation primitives: src_mask and dst_mask. These are
  1754. populated as defined by the [nf|sf|pm|u]acctd_net directive:
  1755. NetFlow/sFlow protocols, BGP, Network files (networks_file) or
  1756. static (networks_mask) being valid data sources.
  1757. + A generic tunnel inspection infrastructure has been developed to
  1758. benefit both pmacctd and uacctd daemons. Handlers are defined via
  1759. configuration file. Once enabled daemons will account basing upon
  1760. tunnelled headers rather than the envelope. Currently the only
  1761. supported tunnel protocol is GTP, the GPRS tunnelling protocol
  1762. (which can be configured as: "tunnel_0: gtp, <UDP port>"). Up to
  1763. 8 different tunnel stacks and up to 4 tunnel layers per stack are
  1764. supported. First matching stack, first matching layer wins.
  1765. + uacctd: support for the MAC layer has been added for the Netlink/
  1766. ULOG Linux packet capturing framework.
  1767. + 'nfprobe_source_ip' feature introduced: it allows to select the
  1768. IPv4/IPv6 address to be used to export NetFlow datagrams to the
  1769. collector.
  1770. + nfprobe, sfprobe: network masks are now exported via NetFlow and
  1771. sFlow. 'pmacctd_net' and its equivalent directives define how to
  1772. populate src_mask and dst_mask values.
  1773. ! cleanup, nfprobe/sfprobe: data source for 'src_as' and 'dst_as'
  1774. primitives is now expected to be always explicitely defined (in
  1775. line with how 'src_net' and 'dst_net' primitives work). See the
  1776. UPGRADE doc for the (limited) backward compatibility impact.
  1777. ! Updated SQL documentation: sql/README.iface guides on 'in_iface'
  1778. and 'out_iface' primitives; sql/README.mask guides on 'src_mask'
  1779. and 'dst_mask' primitives; sql/README.is_symmetric guides on
  1780. 'is_symmetric' primitive.
  1781. ! fix, nfacctd.h: source and destination network masks were twisted
  1782. in the NetFlow v5 export structure definition. Affected releases
  1783. are: 0.12.0rc4 and 0.12.0.
  1784. ! fix, nfprobe_plugin.c: l2_to_flowrec() was missing some variable
  1785. declaration when the package was configured for compilation with
  1786. --disable-l2. Thanks to Brent Van Dussen for reporting the issue.
  1787. ! fix, bgp.c: bgp_attr_munge_as4path() return code was not defined
  1788. for some cases. This was causing some BGP messages to be marked
  1789. as malformed.
  1790. ! fix, sfprobe: a dummy MAC layer was created whenever this was not
  1791. included as part of the captured packet. This behaviour has been
  1792. changed and header protocol is now set to 11 (IPv4) or 12 (IPv6)
  1793. accordingly. Thanks to Neil McKee for pointing the issue.
  1794. ! workaround, building sub-system: PF_RING enabled libpcap was not
  1795. recognized due to missing of pcap_dispatch(). This is now fixed.
  1796. 0.12.0 -- 16-02-2010
  1797. + 'is_symmetric' aggregation primitive has been implemented: aimed
  1798. at easing detection of asymmetric traffic. It's based on rule
  1799. definitions supplied in a 'bgp_is_symmetric_map' map, reloadable
  1800. at runtime.
  1801. + A new 'bgp_daemon_allow_file' configuration directive allows to
  1802. specify IP addresses that can establish a BGP session with the
  1803. collector's BGP thread. Many thanks to Erik van der Burg for
  1804. contributing the idea.
  1805. + 'nfacctd_ext_sampling_rate' and 'sfacctd_ext_sampling_rate' are
  1806. introduced: they flag the daemon that captured traffic is being
  1807. sampled. Useful to tackle corner cases, ie. the sampling rate
  1808. reported by the NetFlow/sFlow agent is missing or incorrect.
  1809. + The 'bgp_follow_nexthop' feature has been extended so that extra
  1810. IPv4/IPv6 prefixes can be supplied. Up to 32 IP prefixes are now
  1811. supported and a warning message is generated whenever a supplied
  1812. string fails parsing.
  1813. + Pre-Tagging: implemented 'src_local_pref' and 'src_comms' keys.
  1814. These allow tagging based on source IP prefix local_pref (sourced
  1815. from either a map or BGP, ie. 'bgp_src_local_pref_type: map',
  1816. 'bgp_src_local_pref_type: bgp') and standard BGP communities.
  1817. + Pre-Tagging: 'src_peer_as' key was extended in order to match on
  1818. BGP-sourced data (bgp_peer_src_as_type: bgp).
  1819. + Pre-Tagging: introduced 'comms' key to tag basing on up to 16
  1820. standard BGP communities attached to the destination IP prefix.
  1821. The lookup is done against the BGP RIB of the exporting router.
  1822. Comparisons can be done in either match-any or match-all fashion;
  1823. xidDocumentation and examples updated.
  1824. ! fix, util.c: load_allow_file(), empty allow file was granting a
  1825. connection to everybody being confused with a 'no map' condition.
  1826. Now this case is properly recognized and correctly translates in
  1827. a reject all clause.
  1828. ! fix, sql_common.c: log of NetFlow micro-flows to a SQL database
  1829. (nfacctd_sql_log directive) was not correctly getting committed
  1830. to the backend, when sql_history was disabled.
  1831. ! fix, mysql|pgsql|sqlite_plugin.c: 'flows' aggregation primitive
  1832. was not suitable to mix-and-match with BGP related primitives
  1833. (ie. peer_dst_as, etc.) due to an incorrect check. Many thanks
  1834. to Zenon Mousmoulas for the bug report.
  1835. ! fix, pretag_handlers.c: tagging against NetFlow v9 4-bytes in/out
  1836. interfaces was not working properly. Thanks to Zenon Mousmoulas
  1837. for reporting the issue.
  1838. 0.12.0rc4 -- 21-12-2009
  1839. + BGP-related source primitives are introduced, namely: src_as_path,
  1840. src_std_comm, src_ext_comm, src_local_pref and src_med. These add
  1841. to peer_src_as which was already implemented. All can be resolved
  1842. via reverse BGP lookups; peer_src_as, src_local_pref and src_med
  1843. can also be resolved via lookup maps which support checks like:
  1844. bgp_nexthop (RPF), peer_dst_as (RPF), input interface and source
  1845. MAC address. Many thanks to Zenon Mousmoulas and GRNET for their
  1846. fruitful cooperation.
  1847. + Memory structures to store BGP-related primitives have been
  1848. optimized. Memory is now allocated only for primitives part of
  1849. the selected aggregation profile ('aggregate' config directive).
  1850. + A new 'bgp_follow_nexthop' configuration directive is introduced
  1851. to follow the BGP next-hop up to the edge of the routing domain.
  1852. This is particularly aimed at networks not running MPLS, where
  1853. hop-by-hop routing is in place.
  1854. + Lookup maps for BGP-related source primitives (bgp_src_med_map,
  1855. bgp_peer_src_as_map, bgp_src_local_pref_map): result of check(s)
  1856. can now be the keyword 'bgp', ie. 'id=bgp' which triggers a BGP
  1857. lookup. This is thought to handle exceptions to static mapping.
  1858. + A new 'bgp_peer_as_skip_subas' configuration directive is being
  1859. introduced. When computing peer_src_as and peer_dst_as, returns
  1860. the first ASN which is not part of a BGP confederation; if only
  1861. confederated ASNs are on the AS-Path, the first one is returned
  1862. instead.
  1863. + Pre-Tagging: support has been introduced for NetFlow v9 traffic
  1864. direction (ingress/egress).
  1865. + Network masks part of NetFlow/sFlow export protocols can now be
  1866. used to compute src_net, dst_net and sum_net primitives. As a
  1867. result a set of directives [nfacctd|sfacctd|pmacctd|uacctd]_net
  1868. allows to globally select the method to resolve such primitives,
  1869. valid values being: netflow, sflow, file (networks_file), mask
  1870. (networks_mask) and bgp (bgp_daemon).
  1871. + uacctd: introduced support for input/output interfaces, fetched
  1872. via NetLink/ULOG API; interfaces are available for Pre-Tagging,
  1873. and inclusion in NetFlow and sFlow exports. The implementation
  1874. is courtesy of Stig Thormodsrud.
  1875. + nfprobe, sfprobe: new [nfprobe|sfprobe]_peer_as option to set
  1876. source/destination ASNs, part of the NetFlow and sFlow exports,
  1877. to the peer-AS rather than origin-AS. This feature depends on a
  1878. working BGP daemon thread setup.
  1879. ! A few resource leaks were detected and fixed. Patch is courtesy
  1880. of Eric Sesterhenn.
  1881. ! bgp/bgp.c: thread concurrency was detected upon daemon startup
  1882. under certain conditions. As a solution the BGP thread is being
  1883. granted a time advantage over the traffic collector thread.
  1884. ! bgp/bgp.c: fixed a security issue which could have allowed a
  1885. malicious user to disrupt established working BGP sessions by
  1886. exploiting the implemented concept of BGP session replenishment;
  1887. this has been secured by a check against the session holdtime.
  1888. Many thanks to Erik van der Burg for spotting the issue.
  1889. ! bgp/bgp.c: BGP listener socket now sets SO_REUSEADDR option for
  1890. quicker turn around times while stopping/starting the daemon.
  1891. ! net_aggr.c: default route ( was considered invalid;
  1892. this is now fixed.
  1893. 0.12.0rc3 -- 28-10-2009
  1894. + Support for NetFlow v9 sampling via Option templates and
  1895. data is introduced; this is twofold: a) 'nfacctd_renormalize'
  1896. configuration directive is now able to renormalize NetFlow v9
  1897. data on-the-fly by performing Option templates management; b)
  1898. 'nfprobe', the NetFlow probe plugin, is able to flag sampling
  1899. rate (either internal or external) when exporting flows to the
  1900. collector.
  1901. + '[pm|u]acctd_ext_sampling_rate' directives are introduced to
  1902. support external sampling rate scenarios: packet selection is
  1903. performed by the underlying packect capturing framework, ie.
  1904. ULOG, PF_RING. Making the daemon aware of the sampling rate,
  1905. allows to renormalize or export such information via NetFlow
  1906. or sFlow.
  1907. + pmacctd: the IPv4/IPv6 fragment handler engine was reviewed
  1908. to make it sampling-friendly. The new code hooks get enabled
  1909. when external sampling (pmacctd_ext_sampling_rate) is defined.
  1910. + A new 'uacctd' daemon is added to the set; it is based on the
  1911. Netlink ULOG packet capturing framework; this implies it works
  1912. only on Linux and can be optionally enabled when compling by
  1913. defining the '--enable-ulog' switch. The implementation is
  1914. fully orthogonal with the existing feature set. Thanks very
  1915. much to: A.O. Prokofiev for contributing the original idea
  1916. and code; Stig Thormodsrud for his support and review.
  1917. + The 'tag2' primitive is introduced. Its aim is to support
  1918. traffic matrix scenarios by giving a second field dedicated
  1919. to tag traffic. In a pre_tag_map this can be employed via the
  1920. 'id2' key. See examples in the 'examples/'
  1921. document. SQL plugins write 'tag2' content in the 'agent_id2'
  1922. field. Read 'sql/README.agent_id2' document for reference.
  1923. + Some new directives to control and re-define file attributes
  1924. written by the pmacct daemons, expecially when launched with
  1925. increased priviledges, are introduced: file_umask, files_uid,
  1926. files_gid. Files to which these apply include, ie. pidfile,
  1927. logfile and BGP neighbors file.
  1928. ! fix, bgp/bgp.c: upon reaching bgp_daemon_max_peers threshold,
  1929. logs were flooded by warnings even when messages were coming
  1930. from a previously accepted BGP neighbor. Warnings are now sent
  1931. only when a new BGP connection is refused.
  1932. ! fix, nfprobe/netflow9.c: tags (pre_tag_map, post_tag) were set
  1933. per pair of flows, not respecting their uni-directional nature.
  1934. It was generating hiding of some tags.
  1935. ! fix, nfprobe/netflow9.c: templates were (wrongly) not being
  1936. included in the count of flows sent in NetFlow v9 datagrams.
  1937. While this was not generating any issues with parsing flows,
  1938. it was originating visualization issues in Wireshark.
  1939. ! fix, SQL plugins: CPU hitting 100% has been determined when
  1940. sql_history is disabled but sql_history_roundoff is defined.
  1941. Thanks to Charlie Allom for reporting the issue.
  1942. ! fix, sfacctd.c: input and output interfaces (non-expaneded
  1943. format) were not correcly decoded creating issues to Pre-
  1944. tagging. Thanks to Jussi Sjostrom for reporting the issue.
  1945. 0.12.0rc2 -- 09-09-2009
  1946. + BGP daemon thread has been tied up with both the NetFlow and
  1947. sFlow probe plugins, nfprobe and sfprobe, allowing to encode
  1948. dynamic ASN information (src_as, dst_as) instead of reading
  1949. it from text files. This finds special applicability within
  1950. open-source router solutions.
  1951. + 'bgp_stdcomm_pattern_to_asn' feature is introduced: filters
  1952. BGP standard communities against the supplied pattern. The
  1953. first matching community is split using the ':' symbol. The
  1954. first part is mapped onto the peer AS field while the second
  1955. is mapped onto the origin AS field. The aim is to deal with
  1956. prefixes on the own address space. Ie. BGP standard community
  1957. XXXXX:YYYYY is mapped as: Peer-AS=XXXXX, Origin-AS=YYYYY.
  1958. + 'bgp_neighbors_file' feature is introduced: writes a list of
  1959. the BGP neighbors in the established state to the specified
  1960. file. This gets particularly useful for automation purposes
  1961. (ie. auto-discovery of devices to poll via SNMP).
  1962. + 'bgp_stdcomm_pattern' feature was improved by supporting the
  1963. regex '.' symbol which can be used to wildcard a pre-defined
  1964. number of characters, ie. '65534:64...' will match community
  1965. values in the range 64000-64999 only.
  1966. + SQL preprocess layer: removed dependency between actions and
  1967. checks. Overral logics was reviewed to act more consistently
  1968. with recently introduced SQL cache entry status field.
  1969. + SQL common layer: poll() timeout is now calculated adaptively
  1970. for increased deadline precision.
  1971. + sql_startup_delay feature functionality was improved in order
  1972. to let it work as a sliding window to match NetFlow setups in
  1973. which a) mainain original flow timestamps and b) enable the
  1974. sql_dont_try_update feature is required.
  1975. ! DST (Daylight Saving Time) support introduced to sql_history
  1976. and sql_refresh_time directives. Thanks to <> for
  1977. reporting the issue.
  1978. ! fix, pmacctd.c: initial sfprobe plugin checks were disabling
  1979. IP fragments handler. This was causing pmacctd to crash under
  1980. certain conditions. Thanks to Stig Thormodsrud for having
  1981. reported the issue.
  1982. ! fix, nfprobe, netflow5.c: missing htons() call while encoding
  1983. src_as primitive.
  1984. ! fix, BGP thread, bgp_aspath.c: estimated AS-PATH length was
  1985. not enough for 32-bit ASNs. String length per-ASN increased
  1986. from 5 to 10 chars.
  1987. ! Documentation update, EXAMPLES: how to establish a local BGP
  1988. peering between pmacctd and Quagga 0.99.14 for NetFlow and
  1989. sFlow probe purposes.
  1990. ! fix, print_status_table(): SEGV was showing up while trying
  1991. to retrieve xFlow statistics by sending a SIGUSR1 signal and
  1992. a collector IP address was not configured.
  1993. ! ip_flow.[c|h]: code cleanup.
  1994. 0.12.0rc1 -- 01-08-2009
  1995. + a BGP daemon thread has been integrated in both the NetFlow
  1996. and sFlow collectors, nfacctd and sfacctd. It maintains per-
  1997. peer RIBs and supports MP-BGP (IPv4, IPv6) and 32-bit ASNs.
  1998. As a result the following configuration directives are being
  1999. introduced: bgp_daemon, bgp_daemon_ip, bgp_daemon_max_peers,
  2000. bgp_daemon_port and bgp_daemon_msglog. For a quick-start and
  2001. implementation notes refer to EXAMPLES document and detailed
  2002. configuration directives description in CONFIG-KEYS.
  2003. + A new set of BGP-related aggregation primitives are now
  2004. supported by the "aggregate" directive: std_comm, ext_comm,
  2005. as_path, peer_src_ip, peer_dst_ip, peer_src_as, peer_dst_as,
  2006. med, local_pref. A few extra directives are being introduced
  2007. to support (filter, map, cut down, etc.) some primitives:
  2008. bgp_peer_src_as_type, bgp_peer_src_as_map, bgp_aspath_radius,
  2009. bgp_stdcomm_pattern and bgp_extcomm_pattern.
  2010. + nfacctd_as_new supports a new value "bgp". It is meant to
  2011. populate src_as and dst_as primitives by looking up source
  2012. and destination IP prefixes against the NetFlow (or sFlow)
  2013. agent RIB.
  2014. + A new sql_table_type directive is introduced: by combining
  2015. it with sql_table_version, defines one of the standard BGP
  2016. tables.
  2017. + Two new directives have been developed to support scenarios
  2018. where NetFlow (or sFlow) agents are not running BGP or have
  2019. default-only or partial views: bgp_follow_default and
  2020. bgp_agent_map.
  2021. + 4-bytes ASNs are now supported: including NetFlow and sFlow
  2022. collectors, NetFlow and sFlow probes, networks_file to map
  2023. prefixes to ASNs. The new BGP daemon implementation is, of
  2024. course, fully compliant.
  2025. + Pre-Tagging: the ID is now a 32-bit unsigned value (it was
  2026. 16-bit). As a result, there valid tags can be in the range
  2027. 1-4294967295 and maps can now express the resulting ID as
  2028. an IPv4 address (ie. bgp_agent_map).
  2029. + Pre-tagging: support for 32-bit input/output interfaces is
  2030. now available.
  2031. ! fix, sql_common.c: read_SQLquery_from_file() was returning
  2032. a random value, regardless of the successful result. Patch
  2033. has been provided provided by Giedrius Liubavicius
  2034. ! fix, pmacct.c: when unused, source/destination IP address
  2035. fields were presented as NULL values. This is now replaced
  2036. with a '0' value to improve output parsing.
  2037. ! Standard major release compilation check-pointing: thanks
  2038. very much to Manuel Pata and Tobias Lott for their strong
  2039. support with OpenBSD and FreeBSD respectively.
  2040. 0.11.6 -- 07-04-2009
  2041. + Introduced support for tag ranges into the 'pre_tag_filter'
  2042. configuration directive (ie. '10-20' matches traffic tagged
  2043. in the range 10..20). This works both in addition to and in
  2044. combination with negations.
  2045. + Tcpdump-style filters, ie. 'aggregate_filter', now support
  2046. indexing within a packet, ie. 'ether[12:2]', to allow a more
  2047. flexible separation of the traffic.
  2048. + Introduced support for descriptions in networks definition
  2049. files pointed by the 'networks_file' configuration directive.
  2050. Thanks to Karl O. Pinc for contributing the patch.
  2051. ! fix, pmacctd: libpcap DLT_LINUX_SLL type is not defined in
  2052. older versions of the library. It was preventing successful
  2053. compilation of pmacct on OpenBSD. This has been fixed by
  2054. defining internally to pmacct all DLT types in use. Thanks
  2055. to Karl O. Pinc for his support.
  2056. ! fix, IPv6 networks_file, load_networks6(): wrong masks were
  2057. applied to IPv6 networks due to dirty temporary buffers for
  2058. storing IPv6 addresses and masks. Short '::' IPv6 format is
  2059. currently not supported. Thanks to Robert Blechinger for
  2060. flagging the issue.
  2061. ! fix, pretag.c: Pre-Tagging infrastructure was SEGV'ing after
  2062. having been instructed to reload via a SIGHUP signal. Patch
  2063. is courtesy of Denis Cavrois and the Acipia development team.
  2064. ! fix, sfacctd, nfacctd: Assign16() was not handling correctly
  2065. 2-bytes EtherType values (ie. 0x86dd, 0x8847) in 802.1Q tags.
  2066. As a result 'aggregate_filter' was not able to correctly match
  2067. IPv6-related filters. Thanks to Axel Apitz for reporting the
  2068. issue.
  2069. ! fix, xflow_status.c: a cosmetic bug was displaying sequence
  2070. numbers without applying previous increment. This definitely
  2071. will help troubleshooting and debugging.
  2072. ! fix, sfacctd, sfv245_check_status(): AF of the sFlow agent
  2073. is now explicitely defined: when IPv6 is enabled the remote
  2074. peer address can be reported as IPv4-mapped IPv6 address. This
  2075. was causing warning messages to report the wrong sFlow agent
  2076. IP address. Thanks to Axel Apitz for reporting the issue.
  2077. ! fix, IMT plugin was crashing upon receipt of a classification
  2078. table request (WANT_CLASS_TABLE) when stream classification
  2079. was actually disabled.
  2080. ! fix, pmacct.c: classifier index was not brought back to zero
  2081. by the pmacct client. This was preventing the client to show
  2082. correct stream classification when it was feeded with multiple
  2083. queries. The fix is courtesy of Fabio Cairo.
  2084. ! fix, MySQL plugin: upon enabling of the 'nfacctd_sql_log'
  2085. directive, 'stamp_updated' field was incorrectly reported
  2086. as '0000-00-00 00:00:00' due to wrong field formatting. Thanks
  2087. to Brett D'Arcy for reporting and patching the issue.
  2088. ! Initial effort to clean the code up by strcpy() calls. Thanks
  2089. to Karl O. Pinc for taking such initiative.
  2090. 0.11.5 -- 21-07-2008
  2091. + SQL UPDATE queries code has been rewritten for increased
  2092. flexibility. The SET statement is now a vector and part of
  2093. it has been shifted into the sql_compose_static_set() routine
  2094. in the common SQL layer.
  2095. + A new sql_locking_style directive is now supported in the
  2096. MySQL plugin. To exploit it, an underlying InnoDB table is
  2097. mandatory. Thanks to Matt Gillespie for his tests.
  2098. + Support for Endace DAG cards is now available; this has been
  2099. tested against libDAG 3.0.0. Many thanks to Robert Blechinger
  2100. for his extensive support.
  2101. + pmacctd, the Linux Cooked device (DLT_LINUX_SLL) handler has
  2102. been enhanced by supporting 'src_mac' and 'vlan' aggregation
  2103. primitives.
  2104. ! fix, xflow_status.c: NetFlow/sFlow collector's IP address is
  2105. being rewritten as when NULL. Was causing SEGVs on
  2106. Solaris/sparc.
  2107. ! fix, server.c: WANT_RESET is copied in order to avoid losing
  2108. it when handling long queries and need to fragment the reply.
  2109. Thanks very much to Ruben Laban for his support.
  2110. ! fix, MySQL plugin: the table name is now escaped in order to
  2111. not conflict with reserved words, if one of those is selected.
  2112. Thanks to Marcel Hecko for reporting the bug.
  2113. ! An extra security check is being introduced in sfacctd as an
  2114. unsupported extension sent over by a Foundry Bigiron 4000 kit
  2115. was causing SEGV issues. Many Thanks to Michael Hoffrath for
  2116. the strong support provided.
  2117. ! fix, 'nfprobe' plugin: AS numbers were not correctly exported
  2118. to the collector when pmacctd was in use. Patch is courtesy of
  2119. Emerson Pinter.
  2120. ! fix, 'nfprobe' plugin: MACs were not properly encapsulated
  2121. resulting in wrong addresses being exported through NetFlow
  2122. v9. The patch is courtesy of Alexander Bergolth.
  2123. ! fix, buffers holding MAC address strings throughout the code
  2124. had not enough space to store the trailing zero. The patch is
  2125. courtesy of Alexander Bergolth.
  2126. ! fix, logfile FD was not correctly passed onto active plugins.
  2127. The patch is courtesy of Denis Cavrois.
  2128. ! Missing field type 60 in NetFlow v9 IPv6 flows, was leading
  2129. nfacctd to incorrect flow type selection (IPv4). An additional
  2130. check on the source IP address has now been included to infer
  2131. IPv6 flows. RFC3954 mandates such field type to be present for
  2132. IPv6 flows. The issue has been verified against a Cisco 7600
  2133. w/ RSP720. Many thanks to Robert Blechinger for his extensive
  2134. support.
  2135. 0.11.4 -- 25-04-2007
  2136. + support for TCP flags has been introduced. Flags are ORed on a
  2137. per-aggregate basis (same as what NetFlow does on a per-flow
  2138. basis). The 'aggregate' directive now supports the 'tcpflags'
  2139. keyword. SQL tables v7 have also been introduced in order to
  2140. support the feature inside the SQL plugins.
  2141. + 'nfacctd_sql_log' directive is being introduced. In nfacctd,
  2142. it makes SQL plugins to use a) NetFlow's First Switched value
  2143. as "stamp_inserted" timestamp and b) Last Switched value as
  2144. "stamp_updated" timestamp. Then, a) by not aggregating flows
  2145. and b) not making use of timeslots, this directive allows to
  2146. log singular flows in the SQL database.
  2147. + sfprobe and nfprobe plugins are now able to propagate tags to
  2148. remote collectors through sFlow v5 and NetFlow v9 protocols.
  2149. The 'tag' key must be appended to sfprobe/nfprobe 'aggregate'
  2150. config directives.
  2151. + pmacct memory client is now able to output either TopN bytes,
  2152. flows or packets statistics. The feature is enabled by a new
  2153. '-T' commandline switch.
  2154. + The Pre-Tagging map is now dynamically allocated and a new
  2155. 'pre_tag_map_entries' config directive allows to set the size
  2156. of the map. Its default value (384) should be suitable for
  2157. most common scenarios.
  2158. ! Bugfix in nfprobe plugin: struct cb_ctxt was not initialized
  2159. thus causing the application to exit prematurely (thinking it
  2160. finished available memory). Thanks to Elio Eraseo for fixing
  2161. the issue.
  2162. ! Some misplaced defines were preventing 0.11.3 code to compile
  2163. smoothly on OpenBSD boxes. Thanks to Dmitry Moshkov for fixing
  2164. it.
  2165. ! Bugfix in SQL handlers, MY_count_ip_proto_handler(): an array
  2166. boundary was not properly checked and could cause the daemon
  2167. to SEGV receiving certain packets. Thanks to Dmitry Frolov for
  2168. debugging and fixing the issue.
  2169. ! NF_counters_renormalize_handler() renormalizes sampled NetFlow
  2170. v5 flows. It now checks whether a positive Sampling Rate value
  2171. is defined rather than looking for the Sampling Mode. It makes
  2172. the feature working on Juniper routers. Thanks once again to
  2173. Inge Bjornvall Arnesen.
  2174. 0.11.3 -- 31-01-2007
  2175. + 'aggregate_filter' directive now supports multiple pcap-style
  2176. filters, comma separated. This, in turn, allows to bind up to
  2177. 128 filters to each activated plugin.
  2178. + nfacctd and sfacctd turn-back time when restarting the daemon
  2179. has been significantly improved by both creating new listening
  2180. sockets with SO_REUSEADDR option and disassociating them first
  2181. thing on receiving SIGINT signal.
  2182. + A new threaded version of pmacctd stream classification engine
  2183. is being introduced. Code status is experimental and disabled
  2184. by default; it could be enabled by providing --enable-threads
  2185. at configure time. Many thanks to Francois Deppierraz and Eneo
  2186. Tecnologia for contributing this useful piece of code.
  2187. + A new 'flow_handling_threads' configuration directive allows to
  2188. set the number of threads of the stream classification engine,
  2189. by default 10.
  2190. + A couple new '[ns]facctd_disable_checks' config directives aim
  2191. to disable health checks over incoming NetFlow/sFlow streams
  2192. (ie. in cases of non-standard vendor's implementations). Many
  2193. thanks to Andrey Chernomyrdin for his patch.
  2194. ! sfv245_check_status() was running checks (ie. verify sequence
  2195. numbers) using sender's IP address. More correctly, it has to
  2196. look at the Agent Address field included in sFlow datagrams.
  2197. Many thanks to Juraj Sucik for spotting the issue.
  2198. ! nfprobe plugin was not compiling properly in conjunction with
  2199. --disable-l2 configure switch. Many thanks to Inge Bjornvall
  2200. Arnesen for submitting the patch.
  2201. ! sfacctd: fixed a bug which was preventing 'aggregate_filter'
  2202. to match values properly in src_port, dst_port, ip proto and
  2203. tos fields. Thanks to Chris Fletcher for spotting the issue.
  2204. ! SQL cache: fixed a bug preventing safe actions to take place
  2205. correctly. It has arisen in version 0.11.2 and hadn't severe
  2206. impact.
  2207. 0.11.2 -- 28-11-2006
  2208. + 'sql_max_writers' configuration directive is being introduced:
  2209. sets the maximum number of concurrent writer processes the SQL
  2210. plugin can fire, allowing the daemon to degrade gracefully in
  2211. case of major database unavailibility.
  2212. + 'sql_history_since_epoch' is being introduced: enables the use
  2213. of timestamps (stamp_inserted, stamp_updated) in the standard
  2214. seconds since the Epoch format as an alternative to the default
  2215. date-time format.
  2216. + 'sql_aggressive_classification' behaviour is changed: simpler
  2217. more effective. It now operates by delaying cache-to-DB purge
  2218. of unknown traffic streams - which would still have chances to
  2219. be correctly classified - for a few 'sql_refresh_time' slots.
  2220. The old mechanism was making use of negative UPDATE queries.
  2221. + The way SQL writer processes are spawned by the SQL plugin has
  2222. slightly changed in order to better exploit fork()'s copy-on-
  2223. write behaviour: the writer now is mostly read-only while the
  2224. plugin does most write operations before spawning the writer.
  2225. ! The list of environment variables passed to the SQL triggers,
  2226. 'sql_trigger_exec', has been updated.
  2227. ! Fixed a bug related to sequence number checks for NetFlow v5
  2228. datagrams. Thanks very much to Peter Nixon for reporting it.
  2229. 0.11.1 -- 25-10-2006
  2230. + PostgreSQL plugin: 'sql_use_copy' configuration directive has
  2231. been introduced; instructs the plugin to build non-UPDATE SQL
  2232. queries using COPY (in place of INSERT). While providing same
  2233. functionalities of INSERT, COPY is more efficient. It requires
  2234. 'sql_dont_try_update' to be enabled. Thanks to Arturas Lapiene
  2235. for his support during the development.
  2236. + nfprobe plugin: support for IPv4 ToS/DSCP, IPv6 CoS and MPLS
  2237. top-most label has been introduced.
  2238. ! Some alignment issues concerning both pkt_extras structure and
  2239. Core process to Plugins memory rings have been fixed. Daemons
  2240. are now reported to be running ok on MIPS/SPARC architectures.
  2241. Many thanks to Michal Krzysztofowicz for his strong support.
  2242. ! sfprobe plugin: a maximum default limit of 256 bytes is set
  2243. on packet payload copy when building Flow Samples in pmacctd
  2244. (ie. if capturing full packets through libpcap, we don't want
  2245. them to be entirely copied into sFlow datagrams).
  2246. ! Sanity checks now take place when processing 'sql_refresh_time'
  2247. values and error messages are thrown out.
  2248. ! Fixes have been committed to IPv6 code in xflow_status.c as it
  2249. was not compiling properly on both Solaris and IRIX.
  2250. 0.11.0 -- 27-09-2006
  2251. + NetFlow v5 sampling and renormalization are now supported:
  2252. a) 'nfacctd' is able to renormalize bytes/packets counters and
  2253. apply Pre-Tagging basing on the sampling rate specified in the
  2254. datagram; b) 'sampling_rate' config key applies to 'nfprobe'
  2255. plugin which is now able to generate sampling informations.
  2256. + 'nfacctd' and 'sfacctd' are now able to give out informations
  2257. about the status of active NetFlow/sFlow streams in terms of
  2258. good/bad/missing datagrams. Whenever an anomaly happens (ie.
  2259. missing or bad packets) a detailed message is logged; overral
  2260. reports are logged by sending SIGUSR1 signals to the daemon.
  2261. + 'logfile' configuration directive is introduced: it allows to
  2262. log directly to custom files. This adds to console and syslog
  2263. logging options.
  2264. ! Old renormalization structure, renorm_table, has been dropped;
  2265. the new one, which applies to both NetFlow and sFlow, is tied
  2266. into the brand new xflow_status_table structure.
  2267. ! When 'nfacctd_as_new' was not in use, NetFlow v5 src_as/dst_as
  2268. values were erroneously swapped. Thanks to Thomas Stegbauer
  2269. for reporting the bug.
  2270. ! Incorrect timeout value for poll() has been fixed in 'sfprobe'
  2271. plugin. It was leading the plugin to take too much resources.
  2272. ! 'nfprobe' plugin was inserting jumps while generating sequence
  2273. numbers.
  2274. ! 'nfprobe' plugin behaviour in handling 'networks_file' content
  2275. has been changed and now equals 'sfprobe': IP addresses which
  2276. are not belonging to known networks/ASNs are no longer zeroed.
  2277. ! 'sfprobe' was not generating correct sample_pool values.
  2278. 0.11.0rc3 -- 30-08-2006
  2279. + 'sfprobe' plugin can now transport packet/flow classification
  2280. tags inside sFlow v5 datagrams. Then, such tags can be read by
  2281. the sFlow collector, sfacctd.
  2282. + 'sfprobe' plugin is able to encapsulate basic Extended Gateway
  2283. informations (src_as, dst_as) into sFlow v5 datagrams starting
  2284. from a Networks File - networks_file configuration directive.
  2285. + 'nfprobe' now supports network data coming from libpcap/tcpdump
  2286. style savefile ('pcap_savefile', -I).
  2287. + pmacctd is now able to capture packets from DLT_NULL, which is
  2288. BSD loopback encapsulation link type. Thanks to Gert Burger for
  2289. his support.
  2290. + Sampling layer has been improved: it's now able to sample flows
  2291. from NetFlow datagrams (not only packets arriving through sFlow
  2292. or libpcap); 'sfprobe' sampling layer has been tied into this
  2293. mechanism and as a result, 'sfprobe_sampling_rate' is now an
  2294. alias for 'sampling_rate' and its default value is 1 (ie. no
  2295. sampling). This change will benefit 'sfprobe' in terms of better
  2296. efficiency.
  2297. + A new 'pmacctd_flow_buffer_buckets' directive defines the number
  2298. of buckets of the Flow Buffer. This value has to scale to higher
  2299. power of 2 accordingly to the link traffic rate and is useful
  2300. when packet classification is enabled. Many thanks for testing,
  2301. debugging and support go to Steve Cliffe.
  2302. + A new 'sql_locking_style' directive allows to choose among two
  2303. types of locking: "table" (default) and "row". More details are
  2304. in the CONFIG-KEYS document. "row" locking has to be considered
  2305. as experimental. Many thanks go to Aaron Glenn and Peter Nixon
  2306. for their close support, work and thoughts.
  2307. ! IPv6 support is now working; it was broken in 0.11.0rc2; thanks
  2308. to Nigel Roberts for signalling and fixing the issue.
  2309. ! Fixed a few issues concerning the building system and related to
  2310. the introduction of some new subtrees. Thanks to Kirill Ponomarew
  2311. and Peter Nixon for signalling them.
  2312. ! Fixed some signal()-related issues when running the package under
  2313. DragonflyBSD. Being fork of FreeBSD 4.x, it needs same cautions.
  2314. Thanks to Aaron Glenn for his support.
  2315. 0.11.0rc2 -- 08-08-2006
  2316. + 'nfprobe' plugin can now transport packet/flow classification
  2317. tags inside NetFlow v9 datagrams, using custom field type 200.
  2318. Then, such tags can be read by the NetFlow collector, nfacctd.
  2319. + 'nfprobe' plugin has now ability to select a Engine Type/Engine
  2320. ID through a newly introduced 'nfprobe_engine' config directive.
  2321. It will mainly allow a collector to distinguish between distinct
  2322. probe instances originating from the same IP address.
  2323. + 'nfprobe' plugin now can automagically select different NetFlow
  2324. v9 template IDs, useful when multiple 'nfprobe' plugins run as
  2325. part of the same daemon instance.
  2326. + 'sfprobe' plugin is now able to redistribute NetFlow flows into
  2327. sFlow samples. This adds to sFlow -> sFlow and libpcap -> sFlow.
  2328. + A new data structure to pass extended data to specific plugins
  2329. has been added. It is placed on the ring, next to pkt_data. It
  2330. is meant to pass extra data to plugins and, same time, avoiding
  2331. to inflate the main data structure.
  2332. ! Wrong arguments were injected into a recently introduced Log()
  2333. call in plugin_hooks.c; it's now fixed: under certain conditions,
  2334. this was generating SEGV at startup while using 'sfprobe' plugin.
  2335. ! Updated documentation; examples and quickstart guides for using
  2336. pmacct as both emitter and collector of NetFlow and sFlow have
  2337. been added.
  2338. - Hooks to compile pmacct the no-mmap() style have been removed.
  2339. 0.11.0rc1 -- 20-07-2006
  2341. a new 'nfprobe' plugin is available and allows to create NetFlow
  2342. v1/v5/v9 datagrams and export them to a IPv4/IPv6 collector. The
  2343. work is based on softflowd 0.9.7 software. A set of configuration
  2344. directives allows to tune timeouts (nfprobe_timeouts), cache size
  2345. (nfprobe_maxflows), collector parameters (nfprobe_receiver), TTL
  2346. value (nfprobe_hoplimit) and NetFlow version of the datagrams to
  2347. be exported (nfprobe_version). Many thanks to Ivan A. Beveridge,
  2348. Peter Nixon and Sven Anderson for their support and thoughts and
  2349. to Damien Miller, author of softflowd.
  2351. a new 'sfprobe' plugin is available and allows to create sFlow
  2352. v5 datagrams and export them to a IPv4 collector. The work is
  2353. based on InMon sFlow Agent 5.6 software. A set of configuration
  2354. directives allows to tune sampling rate (sfprobe_sampling_rate),
  2355. sFlow agent IP address (sfprobe_agentip), collector parameters
  2356. (sfprobe_receiver) and agentSubId value (sfprobe_agentsubid).
  2357. Many thanks to InMon for their software and Ivan A. Beveridge
  2358. for his support.
  2359. ! An incorrect pointer to the received packet was preventing Pre-
  2360. Tagging filters to work correctly against DLT_LINUX_SLL links.
  2361. Many thanks to Zhuang Yuyao for reporting the issue.
  2362. ! Proper checks on protocol number were missing in pmacct client
  2363. program, allowing to look further the bounds of the _protocols
  2364. array. Many thanks to Denis N. Voituk for patching the issue.
  2365. 0.10.3 -- 21-06-2006
  2366. + New Pre-Tagging key 'label': mark the rule with label's value.
  2367. Labels don't need to be unique: when jumping, the first matching
  2368. label wins.
  2369. + New Pre-Tagging key 'jeq': Jump on EQual. Jumps to the supplied
  2370. label in case of rule match. Before jumping, the tagged flow is
  2371. returned to active plugins, as it happens for any regular match
  2372. (set return=false to change this). In case of multiple matches
  2373. for a signle flow, plugins showing 'tag' key inside 'aggregate'
  2374. directive will receive each tagged copy; plugins not receiving
  2375. tags will still receive unique copy of the flow.
  2376. sFlow and NetFlow are usually uni-directional, ie. ingress-only
  2377. or egress-only (to avoid duplicates). Meaningful application of
  2378. JEQs is tagging flows two times: by incoming interface and by
  2379. outgoing one. Only forward jumps are allowed. "next" is reserved
  2380. label and causes to jump to the next rule. Many thanks to Aaron
  2381. Glenn for brainstormings about this point.
  2382. + New Pre-Tagging key 'return': if set to 'true' (which is default
  2383. behaviour) returns the current packet/flow to active plugins, in
  2384. case of match. If switched to 'false', it will prevent this to
  2385. happen. It might be thought either as an extra filtering layer
  2386. (bound to explicit Pre-Tagging rules) or (also in conjunction with
  2387. 'stack') as a way to add flexibility to JEQs.
  2388. + New Pre-Tagging key 'stack': actually '+' (ie. sum symbol) is the
  2389. unique supported value. This key makes sense only if JEQs are in
  2390. use. When matching, accumulate IDs, using the specified operator/
  2391. function. For example, usually <tag>=<currentID>. By setting
  2392. 'stack=+' you will be able to get <tag>=<previousID + currentID>.
  2393. ! Pre-Tagging table now supports a maximum of 384 rules. Because
  2394. of the newly introduced flow alteration features, tables are
  2395. no longer internally re-ordered. However, IPv4 and IPv6 stacks
  2396. are still segregated each other.
  2397. 0.10.2 -- 16-05-2006
  2398. + A new '-l' option is supported by pmacct client tool: it allows
  2399. to enable locking of the memory table explicitely, when serving
  2400. the requested operation.
  2401. + Pre-Tagging infrastructure is now featuring negations for almost
  2402. all supported keys with the exclusion of id, ip and filter. To
  2403. negate, the '-' (minus symbol) need to be prepended; eg.: id=X
  2404. ip=Y in=-1 means tag with X, data received from Net/sFlow agent
  2405. with IP address Y and not coming from interface 1.
  2406. + pre_tag_filter config directive is now featuring same negation
  2407. capabilities as Pre-Tagging infrastructure.
  2408. + Q16 added to FAQS document: a sum of tips for running smoothly
  2409. SQL tables. Many thanks to Wim Kerkhoff and Sven Anderson for
  2410. bringing up the points.
  2411. 0.10.1 -- 18-04-2006
  2412. + AS numbers and IP addresses are no more multiplexed into the same
  2413. field. This ends the limitation of being unable to have both data
  2414. types in the same table (which could be useful for troubleshooting
  2415. purposes, for example). A new SQL table version, v6, is introduced
  2416. in order to support this new data model in all SQL plugins.
  2417. ! Minor fixes to PostgreSQL table schemas, v2 to v5: a) the 'vlan'
  2418. field was erroneously missing from primary keys, slowing down
  2419. INSERT and UPDATE queries; b) primary keys were identified as
  2420. 'acct_pk', thus not allowing multiple tables of different version
  2421. to share the same database; now constraint name is: 'acct_vX_pk',
  2422. with X being the version number. Many thanks to Sven Anderson for
  2423. catching the a)
  2424. ! An alignment issue has been catched when the etheraddr_string()
  2425. gets called from count_src|dst_mac_handlers() in sql_handlers.c
  2426. This seems to be closely connected to a similar trouble catched
  2427. by Daniel Streicher on x86_64 recently.
  2428. ! Fixed an issue with mask_elem() in server.c . Both src|dst_net
  2429. primitives were not (positively, ie. copied back when required)
  2430. masked.
  2431. 0.10.0 -- 22-03-2006
  2432. + Collectors (ie. pmacctd) are now compiled exporting full Dynamic
  2433. Symbol Table. This allows shared object (SO) classifiers to call
  2434. routines included in the collector code. Moreover, a small set
  2435. of library functions - specifically aimed to deal smoothly with
  2436. the classifiers' table - are now included in the collector code:
  2437. pmct_un|register(), pmct_find_first|last_free(), pmct_isfree(),
  2438. pmct_get() and pmct_get_num_entries(). For further reading, take
  2439. a look to README.developers document in classifiers tarball.
  2440. + Classifiers table, which is the linked-list structure containing
  2441. all the active classifiers (RE + SO), is now loaded into a shared
  2442. memory segment, allowing plugins to keep updated about changes to
  2443. the table. Furthermore, the table is now dynamically allocated at
  2444. runtime, allowing an arbitrary number of classifiers to be loaded
  2445. via the new 'classifier_table_num' configuration directive.
  2446. + Pre-Tagging infrastructure adds two new primitives to tag network
  2447. traffic: src_as and dst_as, the source and destination Autonomous
  2448. System Number (ASN). In pmacctd they work against a Network Map
  2449. ('networks_file' configuration directive). In nfacctd and sfacctd
  2450. they work against both sFlow/NetFlow ASN fields and Network Maps.
  2451. Many thanks to Aaron Glenn for his strong support.
  2452. ! PostgreSQL plugin and pmpgplay no more make use of EXCLUSIVE LOCKS
  2453. whenever the sql_dont_try_update directive is activated. We assume
  2454. there is no need for them in a INSERTs-only framework as integrity
  2455. of data is still guaranteed by transactions. The patch has been
  2456. contributed by Jamie Wilkinson, many thanks !
  2457. ! Commandline switches and a configuration file should cohexist and
  2458. the formers need to take precedence over the latter, if required.
  2459. This is a rather standard (and definitely more flexible) approach;
  2460. before this release they were mutual exclusive. Read UPGRADE notes
  2461. at this propo. Thanks for the suggestion to Ivan A. Beveridge.
  2462. ! Some glibc functions (noticeably syslog()) rely upon a rather non-
  2463. standard "extern char *__progname" pointer. Now, its existence is
  2464. properly checked at configuration time. On Linux, setproctitle()
  2465. was causing plugin name/type to get cutted down in messages sent
  2466. to the syslog facility. Thanks to Karl Latiss for his bug report.
  2467. ! Solved a bug involving the load of IPv6 entries from Networks Maps.
  2468. It was causing the count of such entries to be always zero.
  2469. 0.10.0rc3 -- 01-03-2006
  2470. + Aapplication layer (L7) classification capabilities of pmacctd have
  2471. been improved: shared object (SO) classifiers have been introduced;
  2472. they are loaded runtime through dlopen(). pmacct offers them support
  2473. for contexts (informations gathered - by the same classifier - from
  2474. previous packets either in the same uni-directional flow or in the
  2475. reverse one), private memory areas and lower layer header pointers,
  2476. resulting in extra flexibility. Some examples can be found at the
  2477. webpage:
  2478. + 'classifier_tentatives' configuration key has been added: it allows
  2479. to customize the number of tentatives made in order to classify a
  2480. flow. The default number is five, which has proven to be ok but for
  2481. certain types of classification it might result restrictive.
  2482. + 'pmacctd_conntrack_buffer_size' configuration key has been added: it
  2483. (intuitively) defines the size for the connection tracking buffer.
  2484. + Support for Token Ring (IEEE 802.5) interfaces has been introduced
  2485. in pmacctd. Many thanks to Flavio Piccolo for his strong support.
  2486. + 'savefile_wait' (-W commandline) configuration key has been added: if
  2487. set to true causes pmacctd to not return but wait to be killed after
  2488. being finished with the supplied savefile. Useful when pushing data
  2489. from a tcpdump/ethereal tracefile into a memory table (ie. to build
  2490. graphs).
  2491. ! An erroneous replacement of dst with src in mask_elem() was causing
  2492. queries like "pmacct -c dst_host -M|-N <IP address>" to return zero
  2493. counters. Thanks to Ryan Sleevi for signalling the weird behaviour.
  2494. ! Management of the connection tracking buffer has been changed: now,
  2495. a successful search frees the matched entry instead of moving it in
  2496. a chain of stale entries, available for quick reuse.
  2497. ! Error logging of SQL plugins has been somewhat improved: now, error
  2498. messages returned by the SQL software are forwarded to sql_db_error()
  2499. This will definitely allow to exit from the obscure crypticism of
  2500. some generic error strings.
  2501. 0.10.0rc2 -- 14-02-2006
  2502. + CONNECTION TRACKING modules has been introduced into pmacctd: they are
  2503. C routines that hint IP address/port couples for upcoming data streams
  2504. as signalled by one of the parties into the control channel whenever
  2505. is not possible to go with a RE classificator. Conntrack modules for
  2506. FTP, SIP and RTSP protocols are included.
  2507. + 'pidfile' directive way of work has been improved: firstly, whenever
  2508. a collector shuts down nicely, it now removes its pidfile. Secondly,
  2509. active plugins now create a pidfile too: it takes the following form:
  2510. <pidfile>-<plugin type>.<plugin name>. Thanks to Ivan A. Beveridge
  2511. for sharing his thoughts at this propo.
  2512. ! Minor fixes to the classification engine: TCP packets with no payload
  2513. are not considered useful classification tentatives; a new flow can
  2514. inherit the class of his reverse flow whenever it's still reasonably
  2515. valid.
  2516. ! Solved a segmentation fault issue affecting the classificator engine,
  2517. whenever the 'snaplen' directive was not specified. Thanks to Flavio
  2518. Piccolo for signalling it.
  2519. ! Fixed a bug in the PostgreSQL plugin: it appeared in 0.10.0rc1 and was
  2520. uniquely related to the newly introduced negative UPDATE SQL query.
  2521. ! INTERNALS has been updated with few notes about the new classification
  2522. and connection tracking features.
  2523. 0.10.0rc1 -- 24-01-2006
  2524. + PACKET CLASSIFICATION capabilities have been introduced into pmacctd:
  2525. the implemented approach is fully extensible: classification patterns
  2526. are based on regular expressions (RE), human-readable, must be placed
  2527. into a common directory and have a .pat file extension. Many patterns
  2528. for widespread protocols are available at L7-filter project homepage.
  2529. To support this feature, a new 'classifiers' configuration directive
  2530. has been added. It expects full path to a spool directory containing
  2531. the patterns.
  2532. + A new 'sql_aggressive_classification' directive has been added aswell:
  2533. it allows to move unclassified packets even in the case they are no
  2534. more cached by the SQL plugin. This aggressive policy works by firing
  2535. negative UPDATE SQL queries that, whenever successful, are followed
  2536. by positive ones charging the extra packets to their final class.
  2537. ! Input and Output interface fields (Pre-Tagging) have been set to be
  2538. 32 bits wide. While NetFlow is ok with 16 bits, some sFlow agents are
  2539. used to bigger integer values in order to identify their interfaces.
  2540. The fix is courtesy of Aaron Glenn. Thank you.
  2541. ! Flow filtering troubles have been noticed while handling MPLS-tagged
  2542. flows inside NetFlow v9 datagrams. Thanks to Nitzan Tzelniker for his
  2543. cooperation in solving the issue.
  2544. ! A new exit_all() routine now handles nicely fatal errors detected by
  2545. the Core Process, after plugins creation. It avoids leaving orphan
  2546. plugins after the Core Process shutdown.
  2547. 0.9.6 -- 27-Dec-2005
  2548. + Support for 'sql_multi_values' has been introduced into the new SQLite
  2549. 3.x plugin. It allows to chain multiple INSERT queries into a single
  2550. SQL statement. The idea is that inserting many rows at the same time
  2551. is much faster than using separate single-row statements.
  2552. ! MySQL plugin fix: AS numbers were sent to the database unquoted while
  2553. the corresponding field was declared as CHAR. By correctly wrapping AS
  2554. numbers, a major performance increase (expecially when UPDATE queries
  2555. are spawned) has been confirmed. Many thanks to Inge Bjørnvall Arnesen
  2556. for discovering, signalling and solving the issue.
  2557. ! MySQL plugin fix: multi-values INSERT queries have been optimized by
  2558. pushing out of the queue purging loop the proper handling for the EOQ
  2559. event.
  2560. ! The introduction of the intermidiate SQL layer in the 0.9.5 version
  2561. choked the dynamic SQL table creation capability. This has been fixed.
  2562. Thanks to Vitalij Brajchuk for promptly signalling the issue.
  2563. ! The 'pidfile' configuration key has got incorrectly disabled in both
  2564. nfacctd and sfacctd. Thanks to Aaron Glenn for signalling the issue.
  2565. ! The 'daemonize' (-D) configuration key was incorrectly disabling the
  2566. signal handlers from the Core Process once backgrounded. As a result
  2567. the daemon was not listening for incoming SIGINTs. Again, many thanks
  2568. go to Aaron Glenn.
  2569. 0.9.5 -- 07-Dec-2005
  2570. + PMACCT OPENS TO SQLITE 3.x: a fully featured SQLite, version 3.x only,
  2571. plugin has been introduced; SQLite is a small C library that implements
  2572. a self-contained, embeddable, zero-configuration SQL (almost all SQL92)
  2573. database engine. The plugin is LOCK-based and supports the "recovery
  2574. mode" via an alternate database action. Expecially suitable for tiny
  2575. and embedded environments. The plugin can be fired using the keyword
  2576. 'sqlite3'. See CONFIG-KEYS and EXAMPLES for further informations.
  2577. + A new SQL layer - common to MySQL, PostgreSQL and SQLite plugins - has
  2578. been introduced. It's largely callback-based and results in a major
  2579. architectural change: it sits below the specific SQL code (facing the
  2580. Core Process's abstraction layer) and will (hopefully) help in reducing
  2581. potential bugs and will allow for a quick implementation of new SQL
  2582. plugins.
  2583. ! A bug concerning the setup of insert callback functions for summed (in
  2584. + out) IPv6 traffic has been fixed. The issue was affecting all SQL
  2585. plugins.
  2586. ! A bug concerning the handling of MPLS labels has been fixed in pmacctd.
  2587. Many thanks to Gregoire Tourres and Frontier Online for their support.
  2588. 0.9.4p1 -- 14-Nov-2005
  2589. ! Minor bugfix in pretag.c: a wrongly placed memcpy() was preventing the
  2590. code to be compiled by gcc 2.x . Many thanks to Kirill Ponomarew and
  2591. Kris Kennaway for signalling the issue.
  2592. ! Fixed an alignment issue revealed in the query_header structure; it has
  2593. been noticed only under some circumstances: '--enable-64bit' enabled,
  2594. 64bit platform and gcc 3.x . Many thanks to Aaron Glenn for his strong
  2595. support in solving the issue.
  2596. 0.9.4 -- 08-Nov-2005
  2597. + Hot map reload has been introduced. Maps now can be modified and then
  2598. reloaded without having to stop the daemon. SIGUSR2 has been reserved for
  2599. this use. The feature applies to Pre-Tagging map (pre_tag_map), Networks
  2600. map (networks_file) and Ports map (ports_file). It is enabled by default
  2601. and might be disabled via the new 'refresh_maps' configuration directive.
  2602. Further details are in CONFIG-KEYS.
  2603. ! Some major issues have been solved in the processing of libpcap-format
  2604. savefiles. Some output inconsistencies were caused by a corruption of the
  2605. pcap file handler; bufferization is now enabled by default and the last
  2606. buffer is correctly processed. Many thanks go to Amir Plivatsky for his
  2607. strong support.
  2608. ! 'sql_table_schema' directive: in read_SQLquery_from_file() the strchr()
  2609. has been replaced by strrchr() allowing to chain more SQL statements as
  2610. part of the SQL table creation. This results useful, for example, to do
  2611. CREATE INDEX after CREATE TABLE. The patch is courtesy of Dmitriy Nikulin.
  2612. ! SIGTERM signal is now handled properly to ensure a better compatibility
  2613. of all pmacct daemons under the daemontools framework. The patch is
  2614. courtesy of David C. Maple.
  2615. ! Memory plugin: some issues caused by the mix of not compatible compilation
  2616. parameters have been fixed. Now the pmacct client now correctly returns a
  2617. warning message if: counters are of different size (32bit vs 64bit) or IP
  2618. addresses are of different size (IPv4-only vs IPv6-enabled packages).
  2619. ! Print plugin, few bugfixes: the handling of the data ring shared with the
  2620. Core Process was not optimal; it has been rewritten. P_exit() routine was
  2621. not correctly clearing cached data.
  2622. 0.9.3 -- 11-Oct-2005
  2623. + IPv4/IPv6 multicast support has been introduced in the NetFlow (nfacctd)
  2624. and the sFlow (sfacctd) daemons. A maximum of 20 multicast groups may be
  2625. joined by a single daemon instance. Groups can be defined by using the two
  2626. sister configuration keys: nfacctd_mcast_groups and sfacctd_mcast_groups.
  2627. + sfacctd: a new 'sfacctd_renormalize' config key allows to automatically
  2628. renormalize byte/packet counters value basing on informations acquired
  2629. from the sFlow datagram. In particular, it allows to deal with scenarios
  2630. in which multiple interfaces have been configured at different sampling
  2631. rates. It also calculates an effective sampling rate which could differ
  2632. from the configured one - expecially at high rates - because of various
  2633. losses. Such estimated rate is then used for renormalization purposes.
  2634. Many thanks go to Arnaud De-Bermingham and Ovanet for the strong support
  2635. offered during the development.
  2636. + sfacctd: a new 'sampling_rate' keyword is supported into the Pre-Tagging
  2637. layer. It allows to tag aggregates - generated from sFlow datagrams - on
  2638. a sampling rate basis.
  2639. + setproctitle() calls have been introduced (quite conservatively) and are
  2640. actually supported on Linux and BSDs. The process title is rewritten in
  2641. the aim of giving the user more informations about the running processes
  2642. (that is, it's not intended to be just a cosmetic stuff).
  2643. ! sql_preprocess tier was suffering a bug: actions (eg. usrf, adjb), even
  2644. if defined, were totally ignored if no checks were defined aswell. Many
  2645. thanks to Draschl Clemens for signalling the issue.
  2646. ! Some minor bugs have been catched around sfacctd and fixed accordingly.
  2647. Again, many thanks to Arnaud De-Bermingham.
  2648. 0.9.2 -- 14-Sep-2005
  2649. + A new 'usrf' keyword is now supported into the 'sql_preprocess' tier: it
  2650. allows to apply a generic uniform renormalization factor to counters. Its
  2651. use is particularly suitable for use in conjunction with uniform sampling
  2652. methods (for example simple random - e.g. sFlow, 'sampling_rate' directive
  2653. or simple systematic - e.g. sampled NetFlow by Cisco and Juniper).
  2654. + A new 'adjb' keyword is now supported into the 'sql_preprocess' tier: it
  2655. allows to add (or subtract in case of negative value) 'adjb' bytes to the
  2656. bytes counter. This comes useful when fixed lower (link, llc, etc.) layer
  2657. sizes need to be included into the bytes counter (as explained by the Q7
  2658. in the updated FAQS document).
  2659. + A new '--enable-64bit' configuration switch allows to compile the package
  2660. with byte/packet/flow counters of 64bit (instead of the usual 32bit ones).
  2661. ! The sampling algorithm endorsed by the 'sampling_rate' feature has been
  2662. enhanced to a simple randomic one (it was a simple systematic).
  2663. ! Some static memory structures are now declared as constants allowing to
  2664. save memory space (given the multi-process architecture) and offering an
  2665. overral better efficiency. The patch is courtesy of Andreas Mohr. Thanks.
  2666. ! Some noisy compiler warnings have been troubleshooted along with some minor
  2667. code cleanups; the contribution is from Jamie Wilkinson. Thanks.
  2668. ! Some unaligned pointer issues have been solved.
  2669. 0.9.1 -- 16-Aug-2005
  2670. + Probabilistic, flow size dependent sampling has been introduced into the
  2671. 'sql_preprocess' tier via the new 'fss' keyword: it is computed against
  2672. the bytes counter and returns renormalized results. Aggregates which have
  2673. collected more than the 'fss' threshold in the last time window are
  2674. sampled. Those under the threshold are sampled with probability p(bytes).
  2675. For further details read the CONFIG-KEYS and the paper:
  2676. - N.G. Duffield, C. Lund, M. Thorup, "Charging from sampled network usage"
  2678. + Probabilistic sampling under hard resource constraints has been introduced
  2679. into the 'sql_preprocess' tier via the new 'fsrc' keyword: it is computed
  2680. against the bytes counter and returns renormalized results. The method
  2681. selects only 'fsrc' flows from the set of the flows collected during the
  2682. last time window, providing an unbiasied estimate of the real bytes counter.
  2683. For further details read the CONFIG-KEYS and the paper:
  2684. - N.G. Duffield, C. Lund, M. Thorup, "Flow Sampling Under Hard Resource Constraints"
  2686. + A new 'networks_mask' configuration directive has been introduced: it
  2687. allows to specify a network mask - in bits - to be applied apply to src_net
  2688. and dst_net primitives. The mask is applied before evaluating the content of
  2689. 'networks_file' (if any).
  2690. + Added a new signal handler for SIGUSR1 in pmacctd: a 'killall -USR1 pmacctd'
  2691. now returns a few statistics via either console or syslog; the syslog level
  2692. reserved for such purpose is the NOTICE.
  2693. ! sfacctd: an issue regarding non-IP packets has been fixed: some of them
  2694. (mainly ARPs) were incorrectly reported. Now they are properly filtered out.
  2695. ! A minor memory leak has been fixed; it was affecting running instances of
  2696. pmacctd, nfacctd and sfacctd with multiple plugins attached. Now resources
  2697. are properly recollected.
  2698. 0.9.0 -- 25-Jul-2005
  2699. + PMACCT OPENS TO sFlow: support for the sFlow v2/v4/v5 protocol has been
  2700. introduced and a new daemon 'sfacctd' has been added. The implementation
  2701. includes support for BGP, MPLS, VLANs, IPv4, IPv6 along with packet tagging,
  2702. filtering and aggregation capabilities. 'sfacctd' makes use of Flow Samples
  2703. exported by a sFlow agent while Counter Samples are skipped and the MIB is
  2704. ignored. All actually supported backends are available for storage: MySQL,
  2705. PostgreSQL and In-Memory tables.
  2706. lists the network equipments supporting the sFlow protocol.
  2707. + A new commandline option '-L' is now supported by 'nfacctd' and 'sfacctd';
  2708. it allows to specify an IPv4/IPv6 address where to bind the daemon. It is
  2709. the equivalent for the 'nfacctd_ip' and 'sfacctd_ip' configuration directives.
  2710. ! The NetFlow v9 MPLS stack handler has been fixed; it now also sticks the BoS
  2711. bit (Bottom of the Stack) to the last processed label. This makes the flow
  2712. compliant to BPF filters compiled by the newly released libpcap 0.9.3.
  2713. ! Some Tru64 compilation issues related to the ip_flow.[c|h] files have been
  2714. solved.
  2715. ! Some configuration tests have been added; u_intXX_t definitions are tested
  2716. and fixed (whenever possible, ie. uintXX_t types are available). Particularly
  2717. useful on Solaris and IRIX platforms.
  2718. ! Configuration hints for MySQL headers have been enhanced. This will ease the
  2719. compilation of pmacct against MySQL library either from a precompiled binary
  2720. distribution or from the FreeBSD ports. Many hhanks for the bug report go to
  2721. John Von Essen.
  2722. ! NetFlow v8 source/destination AS handlers have been fixed.
  2723. 0.8.8 -- 27-Jun-2005
  2724. + Added IP flows support in pmacctd (release 0.8.5 has seen its introduction
  2725. in nfacctd) for both IPv4 and IPv6 handlers. To enable flows accounting,
  2726. the 'aggregate' directive now supports a new 'flows' keyword. The SQL table
  2727. v4 has to be used in order to support this feature in both SQL plugins.
  2728. + A new 'sum_mac' aggregation method has been added (this is in addition to
  2729. the already consolidated ones: 'sum_host', 'sum_net', 'sum_as', 'sum_port').
  2730. Sum is intended to be the total traffic (inbound traffic summed to outbound
  2731. one) produced by a specific MAC address.
  2732. + Two new configuration directives have been introduced in order to set an
  2733. upper bound to the growth of the fragment (default: 4Mb) and flow (default:
  2734. 16Mb) buffers: 'pmacctd_frag_buffer_size', 'pmacctd_flows_buffer_size'.
  2735. + A new configuration directive 'pmacctd_flow_lifetime' has been added and
  2736. defines how long a flow could remain inactive (ie. no packets belonging to
  2737. such flow are received) before considering it expired (default: 60 secs).
  2738. This is part of the pmacctd IP flows support.
  2739. + Console/syslog feedbacks about either generic errors or malformed packets
  2740. have been greatly enhanced. Along with the cause of the message, now any
  2741. generated message contains either the plugin name/type or the configuration
  2742. file that is causing it.
  2743. ! nfacctd: when IPv6 is enabled (on non-BSD systems) the daemon now listens
  2744. by default on a IPv6 socket getting rid of the v4-in-v6 mapping feature which
  2745. helps in receiving NetFlow datagrams from both IPv4 and IPv6 agents. A new
  2746. configure script switch --enable-v4-mapped is aimed to turn manually on/off
  2747. the feature.
  2748. ! Fixed an issue with the SIGCHLD handling routine on FreeBSD 4.x systems. It
  2749. was causing the sudden creation of zombie processes because of the not correct
  2750. retirement of exited childs. Many thanks for his bug report and strong support
  2751. go to John Von Essen.
  2752. ! Fixed an endianess issue regarding Solaris/x86 platforms caused by not proper
  2753. preprocessor tests. Many thanks to Imre Csatlos for his bug report.
  2754. ! Fixed the default schema for the PostgreSQL table v4. The 'flows' field was
  2755. lacking of the 'DEFAULT 0' modifier; it was causing some troubles expecially
  2756. when such tables were used in conjunction with the 'sql_optimize_clauses'
  2757. directive. Many thanks for his bug report and strong support go to Anik Rahman.
  2758. 0.8.7 -- 14-Jun-2005
  2759. + pmacctd: MPLS support has been introduced. MPLS (on ethernet and ppp links)
  2760. and MPLS-over-VLAN (ethernet only) packets are now supported and passed to
  2761. upper layer routines. Filtering and tagging (Pre-Tagging) packets basing on
  2762. MPLS labels is also supported. Recent libpcap is required (ie, CVS versions
  2763. >= 06-06-2005 are highly adviceable because of the support for MPLS label
  2764. hierarchies like "mpls 100000 and mpls 1024" that will match packets with
  2765. an outer label of 100000 and an inner label of 1024).
  2766. + nfacctd: VLAN and MAC addresses support for NetFlow v9 has been introduced.
  2767. Each of them is mapped to its respective primitive (vlan, src_mac, dst_mac);
  2768. filtering and tagging (Pre-Tagging) IPv4/IPv6 flows basing on them is also
  2769. supported.
  2770. + nfacctd: filtering and tagging (Pre-Tagging) IPv4/IPv6 flows basing on MPLS
  2771. labels has been introduced (read the above notes regarding libpcap version
  2772. requirements).
  2773. + A new packet capturing size option has been added to pmacctd ('snaplen'
  2774. configuration directive; '-L' commandline). It allows to change the default
  2775. portion of the packet captured by the daemon. It results useful to cope
  2776. with not fixed protocol stacks (ie, the MPLS stack).
  2777. + pmacctd: CHDLC support has been introduced. IPv4, IPv6 and MPLS packets are
  2778. supported on this link layer protocol.
  2779. ! Cleanups have been added to the NetFlow packet processing cycle. They are
  2780. mainly aimed to ensure that no stale data is read from circular buffers
  2781. when processing NetFlow v8/v9 packets.
  2782. ! The NetFlow v9 VLAN handling routine was missing a ntohs() call, resulting
  2783. in an ncorrect VLAN id on little endian architectures.
  2784. ! ether_aton()/ether_ntoa() routines were generating segmentation faults on
  2785. x86_64 architectures. They have been replaced by a new handmade couple:
  2786. etheraddr_string()/string_etheraddr(). Many thanks to Daniel Streicher for
  2787. the bug report.
  2788. 0.8.6 -- 23-May-2005
  2789. + The support for dynamic SQL tables has been introduced through the use of
  2790. the following variables in the 'sql_table' directive: %d (the day of the
  2791. month), %H (hours using an 24 hours clock), %m (month number), %M (minutes),
  2792. %w (the day of the week as a decimal number), %W (week number in the current
  2793. year) and %Y (the current year). This enables, for example, substitutions
  2794. like the following ones:
  2795. 'acct_v4_%Y%m%d_%H%M' ==> 'acct_v4_20050519_1500'
  2796. 'acct_v4_%w' ==> 'acct_v4_05'
  2797. + A new 'sql_table_schema' configuration directive has been added in order
  2798. to allow the automatic creation of dynamic tables. It expects as value the
  2799. full pathname to a file containing the schema to be used for table creation.
  2800. An example of the schema follows:
  2801. CREATE TABLE acct_v4_%Y%m%d_%H%M (
  2802. ... PostgreSQL/MySQL specific schema ...
  2803. );
  2804. + Support for MySQL multi-values INSERT clauses has been added. Inserting
  2805. many rows in a single shot has proven to be much faster (many times faster
  2806. in some cases) than using separate single INSERT statements. A new
  2807. 'sql_multi_values' configuration directive has been added to enable this
  2808. feature. Its value is intended to be the size (in bytes) of the multi-values
  2809. buffer. Out of the box, MySQL >= 4.0.x supports values up to 1024000 (1Mb).
  2810. Because it does not require any changes on server side, people using MySQL
  2811. are strongly encouraged to give it a try.
  2812. + A new '--disable-l2' configure option has been added. It is aimed to compile
  2813. pmacct without support for Layer-2 stuff: MAC addresses and VLANs. This
  2814. option - along with some more optimizations to memory structures done in this
  2815. same release - have produced memory savings up to 25% compared to previous
  2816. versions.
  2817. ! Recovery code for PostgreSQL plugin has been slightly revised and fixed.
  2818. 0.8.5 -- 04-May-2005
  2819. + Added IP flows counter support in nfacctd, the NetFlow accounting daemon,
  2820. in addition to the packets and bytes ones. To enable flows accounting, the
  2821. 'aggregate' directive now supports a new 'flows' keyword. A new SQL table
  2822. version, v4, has been also introduced to support this feature in both SQL
  2823. plugins.
  2824. + 'sql_preprocess' directive have been strongly improved by the addition of
  2825. new keywords to handle thresholds. This preprocessing feature is aimed to
  2826. process aggregates (via a comma-separated list of conditionals and checks)
  2827. before they are pulled to the DB, thus resulting in a powerful selection
  2828. tier; whether the check is meet, the aggregate goes on its way to the DB;
  2829. the new thresholds are: maxp (maximum number of packets), maxb (maximum bytes
  2830. transferred), minf/maxf (minimum/maximum number of flows), minbpp/maxbbp
  2831. (minimum/maximum bytes per packet average value), minppf/maxppf (minimum/
  2832. maximum packets per flow average value).
  2833. + Added a new 'sql_preprocess_type' directive; the values allowed are 'any'
  2834. or 'all', with 'any' as default value. It is intended to be the connective
  2835. whether 'sql_preprocess' contains multiple checks. 'any' requires that an
  2836. aggregate has to match just one of the checks in order to be valid; 'all'
  2837. requires a match against all of the checks instead.
  2838. + Added the ability to instruct a BPF filter against the ToS field of a NetFlow
  2839. packet.
  2840. ! Minor optimizations on the 'sql_preprocess' handler chain.
  2841. 0.8.4 -- 14-Apr-2005
  2842. + Added support for NetFlow v7/v8. The Version 7 (v7) format is exclusively
  2843. supported by Cisco Catalyst series switches equipped with a NetFlow feature
  2844. card (NFFC). v7 is not compatible with Cisco routers. The Version 8 (v8)
  2845. format adds (with respect to older v5/v7 versions) router-based aggregation
  2846. schemes.
  2847. + Added the chance to tag packets basing on NetFlow v8 aggregation type field.
  2848. As the keyword suggests, it will work successfully just when processing
  2849. NetFlow v8 packets. Useful to split - backend side - data per aggregation
  2850. type.
  2851. + pmacct client now is able to ask for the '0' (that is, untagged packets) tag
  2852. value. Moreover, all 'sum' aggregations (sum_host, sum_net, sum_as, sum_port)
  2853. can now be associated with both Pre/Post-Tagging.
  2854. ! Fixed a serious memory leak located in the routines for handling NetFlow v9
  2855. templates. While the bug was needing certain conditions to manifest, anyone
  2856. using NetFlow v9 is strongly encouraged to upgrade to this version. All
  2857. previous versions were affected.
  2858. ! Some gcc4 compliance issues have been solved. The source code is known to
  2859. work fine on amd64 architectures. Thanks very much to Marcelo Goes for his
  2860. patch.
  2861. ! Engine Type/Engine ID fields were not correctly evaluated when using NetFlow
  2862. v5 and Pre-Tagging. The issue has been fixed.
  2863. ! Long comments in the Ports Definition File were causing some incorrect error
  2864. messages. However it seems the file were processed correctly. Thanks to Bruno
  2865. Mattarollo for signalling the issue.
  2866. ! Minor fix to plugins hooking code. The reception of sparse SIGCHLD signals
  2867. were causing the poll() to return. The impact was null. The issue has been
  2868. fixed by ignoring such signals.
  2869. 0.8.3 -- 29-Mar-2005
  2870. + Pre-Tagging capabilities have been further enhanced: captured traffic can
  2871. be now marked basing on the NetFlow nexthop/BGP nexthop fields. While the
  2872. old NetFlow versions (v1, v5) carry an unique 'nexthop' field, NetFlow v9
  2873. supports them into two distinguished fields.
  2874. + Packet/flows tagging is now explicit, gaining more flexibility: a new 'tag'
  2875. keyword has been added to the 'aggregate' directive. It causes the traffic
  2876. to be actually marked; the 'pre_tag_map' and 'post_tag' directives now just
  2877. evaluate the tag to be assigned. Read further details about this topic in
  2878. the UPGRADE document.
  2879. + The 'pre_tag_filter' directive now accepts 0 (zero) as valid value: we have
  2880. to remember that zero is not a valid tag; hence, its support allows to split
  2881. or filter untagged traffic from tagged one.
  2882. + Documentation has been expanded: a new FAQS entry now describes few and easy
  2883. tweaks needed to replace the bytes counter type from u_int32_t to u_int64_t
  2884. throughout the code (provided that the OS supports this type); it's useful
  2885. in conjunction with the In-Memory plugin while exposed to very sustained
  2886. traffic loads. A new FAQS entry describes the first efforts aimed to integrate
  2887. pmacctd with popular flow-tools software by the way of the flow-export tool.
  2888. A new UPGRADE document has been also created.
  2889. ! pmacct client was handling counters returned by the '-N' switch as signed
  2890. integers, which is not correct. The issue has been fixed. Many thanks to
  2891. Tobias Bengtsson for signalling it.
  2892. ! Two new routines file_lock()/file_unlock() have replaced the flock() calls
  2893. because they were preventing the pmacct code to compile on Solaris. Basing
  2894. over hints collected at configure time, the routines enable either the flock()
  2895. or fcntl() code. Many thanks to Jan Baumann for signalling and solving the
  2896. issue.
  2897. 0.8.2 -- 08-Mar-2005
  2898. + Pre-Tagging capabilities have been enhanced: now, a Pre Tag Map allows to
  2899. mark either packets or flows basing on the outcome of a BPF filter. Because
  2900. of this new feature, Pre-tagging has been introduced in 'pmacctd' too.
  2901. Pre-tagging was already allowing 'nfacctd' to translate some NetFlow packet
  2902. fields (exporting agent IP address, Input/Output interface, Engine type and
  2903. Engine ID) into an ID (also referred as 'tag'), a small number in the range
  2904. 1-65535.
  2905. + A new 'pmacctd_force_frag_handling' configuration directive has been added;
  2906. it aims to support 'pmacctd' Pre-Tagging operations: whether the BPF filter
  2907. requires tag assignation based on transport layer primitives (e.g. src port
  2908. or dst port), this directive ensures the right tag is stamped to fragmented
  2909. traffic too.
  2910. + Pre Tag filtering (which can be enabled via 'pre_tag_filter' configuration
  2911. directive) allows to filter aggregates basing on the previously evaluated
  2912. ID: whether it matches with at least one of the filter values, the aggregate
  2913. is delivered to the plugin. It has been enhanced by allowing to assign more
  2914. tags to a specific plugin.
  2915. + pmacctd: a new feature to read libpcap savefiles has been added; it can be
  2916. enabled either via the 'pcap_savefile' configuration directive or the '-I'
  2917. commandline switch. Files need to be already closed and correctly finalized
  2918. in order to be read successfully. Many thanks to Rafael Portillo for proposing
  2919. the idea.
  2920. + pmacct client tool supports a new 'tag' keyword as value for the '-c' switch:
  2921. it allows to query the daemon requesting a match against aggregate tags.
  2922. + pmacct client: the behaviour of the '-N' switch (which makes the client to
  2923. return a counter onto the screen suitable for data injection in tools like MRTG,
  2924. Cacti, RRDtool, etc.), has been enhanced: it was already allowing to ask data
  2925. from the daemon but basing only on exact matches. This concept has now extended,
  2926. adding both wildcarding of specific fields and partial matches. Furthermore,
  2927. when multiple requests are encapsulated into a single query, their results are
  2928. by default splitted (that is, each request has its result); a newly introduced
  2929. '-S' switch now allows to sum multiple results into a single counter.
  2930. ! Bugfix: proper checks for the existence of a 'pre_tag_map' file were bypassed
  2931. under certain conditions; however, this erroneous behaviour was not causing any
  2932. serious issue. The correct behaviour is to quit and report the problem to the
  2933. user.
  2934. ! The sampling rate algorithm has been fixed from a minor issue: it was returning
  2935. not expected results when 'sampling_rate: 1'. It now works as expected. Thanks
  2936. to David C. Maple for his extensive support in gaining a better understanding
  2937. of the problem.
  2938. 0.8.1p1 -- 22-Feb-2005
  2939. ! 'sum_host' and 'sum_net' compound primitives have been fixed in order to
  2940. work with IPv6 addresses.
  2941. ! In-Memory Plugin: client queries spotted with both '-r' (reset counters) and
  2942. '-N' (exact match, print counters only) switches enabled were causing the
  2943. daemon to crash whether no entries were found. The problem has been fixed.
  2944. Many thanks to Zach Chambers for signalling the issue.
  2945. ! In-Memory Plugin: client queries spotted with either '-M' or '-N' switches
  2946. enabled were failing to match actual data when either 'sum_host', 'sum_net'
  2947. or 'sum_as' primitives were in use. The issue has been fixed.
  2948. ! The modulo function applied to NetFlow v9 Template Cache has been enhanced
  2949. in order to deal correctly with export agents having an IPv6 address.
  2950. ! Networks/AS definition file: a new check has been added in order to verify
  2951. whether network prefix/network mask pairs are compatible: if they are not,
  2952. the mask is applied to the prefix.
  2953. ! Documentation has been expanded and revised.
  2954. 0.8.1 -- 25-Jan-2005
  2955. + Accounting and aggregation over DSCP, IPv4 ToS field and IPv6 traffic class
  2956. field have been introduced ('aggregate' directive, 'tos' value): these fields
  2957. are actually widely used to implement Layer-3 QoS policies by defining new
  2958. classes of service (most noticeably 'Less than Best Effort' and 'Premium IP').
  2959. MySQL and PostgreSQL tables v3 (third version) have been introduced (they
  2960. contain an additional 4-bytes 'tos' field) to support the new Layer-3 QoS
  2961. accounting.
  2962. + nfacctd core process has been slightly optimized: each flow is encapsulated
  2963. (thus, copied field-by-field) into a BPF-suitable structure only if one or
  2964. more plugins actually require BPF filtering ('aggregate_filter' directive).
  2965. Otherwise, if either filtering is not required or all requested filters fail
  2966. to compile, the copy is skipped.
  2967. + 'pmacct', pmacct client tool: '-e' commandline option (which meaning is:
  2968. full memory table erase) now might be supplied in conjunction with other
  2969. options (thus avoiding the short time delays involved by two consecutive
  2970. queries, ask-then-erase, which may also lead to small losses).
  2971. The new implemented mechanism works as follow: queries over actual data
  2972. (if any) are served before; the table is locked, new aggregates are queued
  2973. until the erasure finishes (it may take seconds if the table is large enough);
  2974. the table is unlocked; the queue of aggregates is processed and all normal
  2975. operations are resumed. Many thanks to Piotr Gackiewicz for the valuable
  2976. exchange of ideas.
  2977. ! Bug fixed in nfacctd: source and destination AS numbers were incorrectly
  2978. read from NetFlow packets. Thanks to Piotr Gackiewicz for his support.
  2979. ! Bug fixed in pmacct client: while retrieving the whole table content was
  2980. displaying espected data, asking just for 'dst_as' field was resulting in
  2981. no results instead. Thanks, once more, to Piotr Gackiewicz.
  2982. 0.8.0 -- 12-Jan-2005
  2983. + PMACCT OPENS TO IPv6: IPv6 support has been introduced in both 'pmacctd'
  2984. and 'nfacctd' daemons. Because it requires larger memory structures to
  2985. store its addresses, IPv6 support has been disabled by default. It could
  2986. be enabled at configure time via '--enable-ipv6' switch. All filtering,
  2987. tagging and mapping functions already support IPv6 addresses. Some notes
  2988. about IPv6 and SQL table schema have been dropped into README.IPv6 file,
  2989. sql section of the tarball.
  2990. + PMACCT OPENS TO NetFlow v9: support for the template-based Cisco NetFlow
  2991. v9 export protocol has been added. NetFlow v1/v5 were already supported.
  2992. 'nfacctd' may now be bound to an IPv6 interface and is able to read both
  2993. IPv4 and IPv6 data flowsets. A single 'nfacctd' instance may read flows
  2994. of different versions and coming from multiple exporting agents. Source
  2995. and destination MAC addresses and VLAN tags are supported in addition to
  2996. the primitives already supported in v1/v5 (source/destination IP addresses,
  2997. AS, ports and IP protocol). Templates are cached and refreshed as soon as
  2998. they are resent by the exporting agent.
  2999. + Pre Tag map ('pre_tag_map' configuration key), which allows to assign a
  3000. small integer (ID) to an incoming flow basing on NetFlow auxiliar data,
  3001. now may apply tags basing also over Engine Type (it provides uniqueness
  3002. with respect to the routing engine on the exporting device) and Engine
  3003. ID (it provides uniqueness with respect to the particular line card or
  3004. VIP on the exporting device) fields. Incoming and Outcoming interfaces
  3005. were already supported. See '' into tarball examples
  3006. section and CONFIG-KEYS document for further details.
  3007. + Raw protocol (DLT_RAW) routine has been added; it usually allows to read
  3008. data from tunnels and sitX devices (used for IPv6-in-IPv4 encapsulation).
  3009. + Some tests for architecture endianess, CPU type and MMU unaligned memory
  3010. access capability have been added. A small and rough (yes, they work the
  3011. hard way) set of unaligned copy functions have been added. They are aimed
  3012. to be introduced through the code, however first tests over MIPS R10000
  3013. and Alpha EV67 (21264A) have shown positive results.
  3014. ! PPPoE and VLAN layer handling routines have been slightly revised for some
  3015. additional checks.
  3016. ! Given the fairly good portability reported from the mmap() code introduced
  3017. through the whole 0.7.x development stage, the use of shared memory segments
  3018. is now enabled by default. The configure switch '--enable-mmap' has been
  3019. replaced by '--disable-mmap'.
  3020. ! 'pmacct' client tool: because of the IPv6 addresses introduction, separator
  3021. character for multiple queries (commandline) have been changed to from
  3022. ':' to ';'.
  3023. ! 'nfacctd': '-F' commandline switch was listed into available options list,
  3024. but getopt() stanza was missing, thus returning an invalid option message.
  3025. Thanks to Chris Koutras for his support in fixing the issue.
  3026. ! Some variable assignations were causing lvalue errors with gcc 4.0. Thanks
  3027. to Andreas Jochens for his support in signalling and solving the problem.
  3028. 0.7.9 -- 21-Dec-2004
  3029. + A new data pre-processor has been introduced in both SQL plugins: it
  3030. allows to filter out data (via conditionals, checks and actions) during
  3031. a cache-to-DB purging event, before building SQL queries; this way, for
  3032. example, aggregates which have accounted just a few packets or bytes may
  3033. be either discarded or saved through the recovery mechanism (if enabled).
  3034. The small set of preprocessing directives is reported into CONFIG-KEYS
  3035. document.
  3036. + Some new environment variables are now available when firing a trigger
  3037. from SQL plugins: $EFFECTIVE_ELEM_NUMBER reports the effective number
  3038. of aggregates (that is, excluding those filtered out at preprocessing
  3039. time) encapsulated in SQL queries; $TOTAL_ELEM_NUMBER reports the total
  3040. number of aggregates instead.
  3041. $INSERT_QUERIES_NUMBER and $UPDATE_QUERIES_NUMBER returns respectively
  3042. the number of aggregates being successfully encapsulated into INSERT
  3043. and UPDATE queries. $ELAPSED_TIME reports the time took to complete
  3044. the last purging event. For further details and the list of supported
  3045. environment variables take a look to TRIGGER_VARS document.
  3046. + Some additions to both logfile players: a new '-n' switch allows to play
  3047. N elements; this way, arbitrary portions of the file may be played using
  3048. '-n' in conjunction with the (already existing) '-o' switch which allows
  3049. to read the logfile starting at a specified offset. New switches '-H',
  3050. '-D', '-T', '-U', '-P' have been introduced to override SQL parameters
  3051. like hostname, DB, table, user and password. The '-t -d' combination
  3052. (test only, debug) now allows to print over the screen the content of
  3053. the logfile.
  3054. + Logfiles size is now limited to a maximum of 2Gb, thus avoiding issues
  3055. connected to the 32bit declaration of off_t. While many OS implment a
  3056. solution to the problem, seems there are few chances to solve it in a
  3057. portable way. When the maximum size is hit the old logfile is rotated
  3058. appending to its filename a trailing small integer ( in a way similar
  3059. to logrotate) and a fresh one is started.
  3060. ! Logfile players: '-s' switch, which was allowing to play one element
  3061. a time, has been superseded. Its current equivalent is: '-n 1'.
  3062. ! The file opening algorithm has been slightly changed in SQL plugins:
  3063. flock() follows shortly the fopen() and all subsequent operations and
  3064. evaluations are thus strictly serialized. freopen() is avoided.
  3065. 0.7.8 -- 02-Dec-2004
  3066. + Recovery logfile structure has been enhanced. Following the logfile
  3067. header has been created a new template structure. Templates will avoid
  3068. the issue of being not able to read old logfiles because of changes to
  3069. internal data structures. Templates are made of an header and a number
  3070. of entries, each describing a single field of the following data.
  3071. Both players, pmmyplay and pmpgplay, are able to parse logfiles basing
  3072. over the template description. Backward logfile compatibility is broken.
  3073. + Execcutable triggering mechanism (from SQL plugins) has been enhanced:
  3074. some status informations (eg. stats of the last purging event) are now
  3075. passed to the trigged executable in the form of environment variables.
  3076. The list of supported variables has been summarized into TRIGGER_VARS
  3077. document. The mechanism allows to spawn executables for post-processsing
  3078. operations at arbitrary timeframes.
  3079. + Support for 'temporary' devices (like PPP and maybe PCMCIA cards too)
  3080. has been introduced. A new configuration directive 'interface_wait' (or
  3081. '-w' commandline) instructs pmacctd to wait for the listening device to
  3082. become available. It works both when in startup phase and when already
  3083. into main loop. A big thanks to Andre Berger for his support.
  3084. ! ppp_handler() routine, which is in charge to handle PPP packets, have
  3085. been totally rewritten. Thanks, again, to Andre Berger for his support.
  3086. ! All link layer handling routines have been revised; some extra checks
  3087. have been added to overcome issues caused from malicious handcrafted
  3088. packets.
  3089. ! Some time handling and timeout issues have been revised into PostgreSQL
  3090. plugin code. They were affecting only the triggering mechanism.
  3091. ! Fixed an execv() bug into MY_Exec() and PG_Exec(). It was causing the
  3092. not correct execution of triggers. Now, a zeroed argv parameter is
  3093. passed to the function. The problem has been verified on FreeBSD.
  3094. 0.7.7 -- 16-Nov-2004
  3095. + Added two new aggregation primitives: 'src_as' and 'dst_as'. They allow
  3096. accounting based over Autonomous System number; 'pmacctd' requires AS
  3097. numbers to be supplied into a 'networks_file' configuration directive
  3098. (which allows to specify the path to a networks definition file);
  3099. 'nfacctd' may either look up AS numbers from the networks definition file
  3100. or read them from each NetFlow flow (this is default). 'nfacctd_as_new'
  3101. key could be used to switch 'nfacctd' behaviour.
  3102. + Added some new aggregation modes: 'sum_net', 'sum_as', 'sum_port' ('sum'
  3103. which is actually an alias for 'sum_host' has been already introduced
  3104. early). Sum is intended to be the total traffic (that is, inbound plus
  3105. outbound traffic amounts) for each entry.
  3106. + Added another aggregation primitive: 'none'. It does not make use of any
  3107. primitive: it allows to see total bytes and packets transferred through
  3108. an interface.
  3109. + The definition of a 'networks_file' enables network lookup: hosts inside
  3110. defined networks are ok; hosts outside them are 'zeroed'. This behaviour
  3111. may now also be applied to 'src_host', 'dst_host' and 'sum_host'. Under
  3112. certain conditions (eg. when using only host/net/as primitives and defined
  3113. networks comprise all transiting hosts) it may be seen an alternative
  3114. way to filter data.
  3115. ! 'frontend'/'backend' PostgreSQL plugin operations have been obsoleted.
  3116. 'unified'/'typed' operations have been introduced instead. See 'sql_data'
  3117. description, CONFIG-KEYS document, for further informations.
  3118. ! Optimizations have been applied to: core process, the newly introduced
  3119. cache code (see 0.7.6) and in-memory table plugin.
  3120. ! Fixed some string handling routines: trim_all_spaces(), mark_columns()
  3121. ! Solved a potential race condition which was affecting write_pid_file()
  3122. 0.7.6 -- 27-Oct-2004
  3123. + Many changes has been introduced on 'pmacct' client side. '-m' switch
  3124. (which output was suitable as MRTG input) has been obsoleted (though it
  3125. will continue to work for next few releases). A new '-N' switch has
  3126. been added: it returns counter value, suitable for integration with
  3127. either RRDtool or MRTG.
  3128. + Support for batch queries have also been added into pmacct client. It
  3129. allows to join up to 4096 requests into a single query. Requests could
  3130. either be concatenated commandline or read from a file (more details are
  3131. in FAQS and EXAMPLES). Batch queries allow to handle efficiently high number
  3132. of requests in a single shot (for example to timely feed data to a large
  3133. amount of graphs).
  3134. + Still pmacct client: '-r' switch, which already allows to reset counters
  3135. for matched entries, now it also applies to group of matches (also referred
  3136. as partial matches).
  3137. + New scripts have been added into the examples tree which show how to
  3138. integrate memory and SQL plugins with RRDtool, MRTG and GNUplot.
  3139. + Memory plugin (IMT) has been further enhanced; each query from pmacct
  3140. client is now evaluated and if involves just a short ride through the
  3141. memory structure, it is served by the plugin itself without spawning a
  3142. new child process. Batch queries support and reordering of fragmented
  3143. queries have also been added.
  3144. + New cache has been introduced in both SQL plugins; its layout is still
  3145. an hash structure but it now features also chains, allocation, reuse and
  3146. retirement of chained nodes. It also sports a LRU list of nodes which eases
  3147. node handling. The new solution avoids the creation of a collision queue,
  3148. ensuring uniqueness of data placed onto the queries queue. While this
  3149. already greatly benefits a directive like 'sql_dont_try_update', it also
  3150. opens new chances for post-processing operations of queries queue.
  3151. 0.7.5 -- 14-Oct-2004
  3152. + Introduced support for the definition of a 'known ports' list, when
  3153. either 'src_port' or 'dst_port' primitives are in use. Known ports
  3154. will get written into the backend; unknown ports will be simply zeroed.
  3155. It could be enabled via 'ports_file' configuration key or '-o' commandline
  3156. switch.
  3157. + Introduced support for weekly and monthly counters breakdown; hourly,
  3158. minutely and daily were already supported. New breakdowns could be
  3159. enabled via 'w' and 'M' words in 'sql_history' and related configuration
  3160. keys.
  3161. + Added a '-i' commandline switch to both 'pmmyplay' and 'pmpgplay' to
  3162. avoid UPDATE SQL queries and skip directly to INSERT ones. Many thanks
  3163. to Jamie Wilkinson.
  3164. ! 'pmmyplay' and 'pmpgplay' code has been optimized and updated; some
  3165. pieces of locking and transactional code were included into the inner
  3166. loop. A big thanks goes to Wim Kerkhoff and Jamie Wilkinson.
  3167. ! Networks aggregation code has been revised and optimized; a direct-mapped
  3168. cache has been introduced to store (and search) last search results
  3169. from the networks table. A binary search algorithm, though optimized,
  3170. over the table has still been preferred over alternative approaches
  3171. (hash, tries).
  3172. 0.7.4 -- 30-Sep-2004
  3173. + Enhanced packet tagging support; it's now broken in Pre-Tagging and
  3174. Post-Tagging; Pre-Tagging allows 'nfacctd' to assign an ID to a flow
  3175. evaluating an arbitrary combination of supported NetFlow packet
  3176. fields (actually: IP address, Input Interface, Output Interface); the
  3177. Pre-Tagging map is global; Pre-Tag is applied as soon as each flow
  3178. is processed; Post-Tagging allows both 'nfacctd' and 'pmacctd' to
  3179. assign an ID to packets using a supplied value; Post-Tagging could be
  3180. either global or local to a single plugin (and more plugins may tag
  3181. differently); Post-Tag is applied as a last action before the packet
  3182. is sent to the plugin. 'nfacctd_id_map' and 'pmacctd_id' configuration
  3183. keys are now obsolete; 'pre_tag_map' and 'post_tag' are introduced to
  3184. replace them.
  3185. + Added support for Pre-Tag filtering; it allows to filter packets basing
  3186. on their Pre-Tag value. The filter is evaluated after Pre-Tagging but
  3187. before Post-Tagging; it adds to BPF filtering support ('aggregate_filter'
  3188. configuration key); 'pre_tag_filter' configuration key is introduced.
  3189. + Added support for Packet Sampling; the current implementation bases on
  3190. a simple systematic algorithm; the new 'sampling_rate' configuration
  3191. key expects a positive integer value >= 1 which is the ratio of the
  3192. packets to be sampled (translates in: pick only 1 out of N packets).
  3193. The key is either global or local (meaning that each plugin could apply
  3194. different sampling rates).
  3195. ! Fixed a bug which was causing crashes in both 'pmacctd' and 'nfacctd'
  3196. when '-r' parameter was specified commandline. Thanks to Ali Nikham
  3197. for his support.
  3198. 0.7.3 -- 31-Aug-2004
  3199. + Added support for both Netflow 'input interface' and 'output interface'
  3200. fields. These two fields are contained in each flow record inside a
  3201. NetFlow packet. It works through ID mapping (read below).
  3202. + The ID map file syntax has been enhanced to allow greater flexibility
  3203. in ID assignation to packets; example: 'id=1 ip= in=3
  3204. out=5'; the above line will cause the 'ID' 1 to be assigned to
  3205. flows exported by a NetFlow agent (for example a router) which IP
  3206. address is '' and transiting from interface '3' to interface
  3207. '5'.
  3208. + In-memory table operations have been enhanced when using shared memory;
  3209. a new reset flag has been added to avoid race conditions.
  3210. ! Configuration lines are no more limited to some fixed maximum length
  3211. but are allocated dynamically; this to overcome the need for long
  3212. configuration lines to declare arbitrary filters and plugin's list.
  3213. Thanks to Jerry Ji for his support.
  3214. ! Configuration handlers, which are responsible to parse and validate
  3215. values for each configuration key, have been rewritten on the way
  3216. for a better portability.
  3217. ! Signal handler routines have been changed to better accomodate SysV
  3218. semantics.
  3219. ! Fixed shared memory mmap() operations on IRIX and SunOS; a further
  3220. test checks for either 'MAP_ANON' or 'MAP_ANONYMOUS' definitions; in
  3221. case of negative outcome, mmap() will use '/dev/zero'.
  3222. ! Packet handlers have been revised and optimized.
  3223. ! Some optimizations have been added when using shared memory; write()
  3224. function has been usually called to signal the arrival of each new packet,
  3225. through the core process/plugin control channel; now it does so if and
  3226. only if the plugin, on the other side, is actually blocking over a poll();
  3227. because of sequence numbers guarantee, data is directly written into
  3228. shared memory segment.
  3229. 0.7.2p1 -- 08-Aug-2004
  3230. ! Multiple fixes in plugin's configuration post checks; negative outcome
  3231. of some checks was leading to clear misbehaviours. Versions affected
  3232. are >= 0.7.0 . A big thanks goes to Alexandra Walford for her support.
  3233. 0.7.2 -- 02-Aug-2004
  3234. + VLAN accounting has been added. The new 'vlan' keyword is supported as
  3235. argument of both '-c' commandline switch and 'aggregate' configuration
  3236. key.
  3237. + Distributed accounting support has been added. It could be enabled into
  3238. 'pmacctd' via 'pmacctd_id' configuration key and into 'nfacctd' via the
  3239. 'nfacctd_id_file' configuration key. While 'pmacctd_id' key expects as
  3240. value a small integer, 'nfacctd_id_file' expects a path to a file which
  3241. contains the mapping: 'IP address of the router (exporting Newflow) ->
  3242. small integer'. This scheme ease tasks such as keeping track of who has
  3243. generated what data and either cluster or keep disjoint data coming from
  3244. different sources when using a SQL database as backend.
  3245. + Introduced SQL table version 2. The SQL schema is the same as existing
  3246. tables with the following additions: support for distributed accounting;
  3247. support for VLAN accounting.
  3248. + Added MAC addresses query capabilties to pmacct client.
  3249. + Added '-r' commandline switch to pmacct client. It can only be used in
  3250. conjunction with '-m' or '-M' switches. It allows to reset packet and
  3251. bytes counters of the retrieved record.
  3252. ! Exit codes have been fixed in both 'pmacctd' and 'nfacctd'. Thanks to
  3253. Jerry Ji for his signallation.
  3254. ! Fixed a problem when retrieving data from memory table: sometimes null
  3255. data (without any error message) was returned to the client; the problem
  3256. has been successfully reproduced only on FreeBSD 5.1: after an accept()
  3257. call, the socket being returned inherits same flags of the listening
  3258. socket, this case non-blocking flag. Thanks to Nicolas Deffayet for his
  3259. support.
  3260. ! Revised PostgreSQL creation script.
  3261. 0.7.1 -- 14-Jul-2004
  3262. + Added shared memory implementation; core process, now, could push data
  3263. into a shared memory segment and then signal arrival of new data to the
  3264. plugin. Shared memory support could be enabled via '--enable-mmap' switch
  3265. at configuration time.
  3266. + Strongly enhanced gathering capabilities of pmacct client; pmacct client
  3267. is used to fetch data from memory plugin; it is, now, able to ask exact
  3268. or partial matches via '-M' switch and return a readable listing output.
  3269. MRTG export capabilities, full table fetch and table status query are
  3270. still supported.
  3271. + Introduced SQL table versioning. It could be enabled via 'sql_table_version'
  3272. configuration switch. It will enable to build new SQL tables (for example
  3273. adding new aggregation methods) while allowing who is not interested in new
  3274. setups to work with old tables.
  3275. + Added checks for packet capture type; informations acquired are later used
  3276. for better handling pcap interface.
  3277. ! Fixed some issues concerning pmacctd VLAN and PPPOE code.
  3278. ! Fixed a mmap() issue on Tru64 systems.
  3279. ! Fixed some minor poll() misbehaviours in MySQL, PgSQL and print plugins;
  3280. they were not correctly handled.
  3281. 0.7.0p1 -- 13-Jul-2004
  3282. ! Fixes in cache code; affects MySQL, PgSQL and print plugins.
  3283. 0.7.0 -- 01-Jul-2004
  3284. + PMACCT OPENS TO NETFLOW: a new network daemon, nfacctd, is introduced:
  3285. nfacctd listens for Netflow V1/V5 packets; is able to apply BPF filters
  3286. and to aggregate packets; it's then able to either save data in a memory
  3287. table, MySQL or PostgreSQL database or simply output packets on the screen.
  3288. It can read timestamps from Netflow packets in msecs, seconds or ignore
  3289. them generating new timestamps; a simple allow table mechanism allows
  3290. to silently discard Netflow packets not generated by a list of trusted
  3291. hosts.
  3292. + Strongly enhanced IP fragmentation handling in pmacctd.
  3293. + Added new checks into the building systems; new hints when it searches
  3294. for libraries and headers; initial tests for C compilers capabilities
  3295. have been added.
  3296. + Works to let pmacct run on IRIX platforms continue; some issues with
  3297. MipsPRO compiler have been solved; added proper compilation flags/hints.
  3298. SIGCHLD is now properly handled and child processes are correctly retired.
  3299. (a thank for his support goes to Joerg Behrens)
  3300. + First, timidous, introduction of mmap() calls in memory plugin; they need
  3301. to be enabled with '--enable-mmap' flag at configure time.
  3302. ! Fixed a potential deadlock issue in PostgreSQL plugin; changed locking
  3303. mechanism. (a big thank to Wim Kerkhoff)
  3304. ! Fixed an issue concerning networks aggregation on Tru64 systems.
  3305. 0.6.4p1 -- 01-Jun-2004
  3306. ! Fixed an issue with cache aliasing in MySQL and PostgreSQL plugins.
  3307. Other plugins are not affected; this potential issue affects only
  3308. version 0.6.4, not previous ones. Anyone using these plugins with
  3309. 0.6.4 is strongly encouraged to upgrade to 0.6.4p1.
  3310. 0.6.4 -- 27-May-2004
  3311. + Added chance to launch executables from both SQL plugins at arbitrary
  3312. time intervals to ease data post-processing tasks. Two new keys are
  3313. available: 'sql_trigger_exec' and 'sql_trigger_time'. If any interval
  3314. is supplied the specified executable is triggered every time data is
  3315. purged from the cache.
  3316. + Added a new 'print' plugin. Enabling it, data is pulled at regular
  3317. intervals to stdout in a way similar to cflowd's 'flow-print'.
  3318. tool. New config keys are 'print_refresh_time', 'print_cache_entries'
  3319. and 'print_markers'. This last key enables the print of start/end
  3320. markers each time the cache is purged.
  3321. + Added 'sql_dont_try_update' switch to avoid UPDATE queries to the DB
  3322. and skip directly to INSERT ones. Performance gains has been noticed
  3323. when UPDATEs are not necessary (eg. when using timeslots to break up
  3324. counters and sql_history = sql_refresh_time).
  3325. Thanks to Jamie Wilkinson.
  3326. + Optimized use of transactions in PostgreSQL plugin; in the new scheme
  3327. is built a single big transaction for each cache purge process. This
  3328. leads to good performance gains; recovery mechanisms have been modified
  3329. to overcome whole transaction trashing. Many thanks to James Gregory
  3330. and Jamie Wilkinson.
  3331. ! Enhanced debug messages output when specific error conditions are returned
  3332. by the DB.
  3333. ! Fixed a potential counters overflow issue in both MySQL and PgSQL
  3334. plugins cache.
  3335. ! Fixed preprocessor definitions issue: LOCK_UN, LOCK_EX are undeclared
  3336. on IRIX and Solaris. Thanks to Wilhelm Greiner for the fix.
  3337. 0.6.3 -- 27-Apr-2004
  3338. + Added support for full libpcap-style filtering capabilities inside
  3339. pmacctd. This allows to bind arbitrary filters to each plugin (in
  3340. addition to already existing chance to apply them to the listening
  3341. interface via 'pcap_filter' configuraiton key). The config key to
  3342. specify these new filters is 'aggregate_filter'.
  3343. + Strongly improved networks definition file handling; now the file is
  3344. parsed and organized as a hierarchical tree in memory. This allows to
  3345. recognize and support networks-in-networks.
  3346. + Initial optimizations has been done over the code produced in last
  3347. few months.
  3348. + Preprocessor definitions has been added to some part of the code, to
  3349. allow pmacctd compile over IRIX. It has been reported to work over a
  3350. IRIX64 6.5.23 box. Thanks to Wilhelm Greiner for his efforts.
  3351. + Added flock() protected access to recovery logfiles.
  3352. ! Fixed an ugly SEGV issue detected in both 0.6.2's logfile player tools.
  3353. 0.6.2 -- 14-Apr-2004
  3354. + Added support for networks aggregation. Two new primitives has
  3355. been added 'src_net' and 'dst_net' to be used in conjunction with
  3356. a network's definitions file (path is supplied via 'networks_file'
  3357. configuration key). An example of this file is in the examples/
  3358. directory.
  3359. When this aggregation is enabled, IP addresses are compared against
  3360. the networks table; then the matching network will get written to
  3361. the backend; if any match occurs a '' is written.
  3362. A really big thank goes to Martin Anderberg for his strong support
  3363. during last weeks.
  3364. + pipe() has been thrown away; socketpair() has been introduced to
  3365. set up a communication channel between pmacctd core process and
  3366. plugins.
  3367. + Added 'plugin_pipe_size' configuration key to adjust queue depth
  3368. (size) beween core process and plugins. A default value is set by
  3369. operating system; it could not suffice when handling heavy traffic
  3370. loads. Added also a specific error string when pipe gets filled.
  3371. + Added 'plugin_buffer_size' configuration key to enable chances to
  3372. bufferize data to be sent to plugins. When under great loads this
  3373. helps in preventing high CPU usage and excessive pressure over
  3374. kernel.
  3375. + SQL plugins aliasing behaviour has been changed; when no free space
  3376. for new data is found and old data has to be pulled out, it's now
  3377. actually written to the DB but it's inserted in a new 'collision
  3378. queue'. This new queue is purged together with the 'queries queue'.
  3379. See INTERNALS for further details.
  3380. + SQL plugins cache behaviour has been changed by a direct-mapped
  3381. one to a 3-ways associative to get better scores when searching
  3382. free space for new data. See INTERNALS for further details.
  3383. + Added 'sql_cache_entries' configuration key to adjust bucket's
  3384. number of SQL plugin cache. As every hashed structure, a prime
  3385. number of buckets is advisable to get better dispersion of data
  3386. through the table.
  3387. ! Fixed a malloc() SEGV issue in in-memory table plugin first
  3388. noticed with gcc 3.3.3 (Debian 20040320) and glibc 2.3.2.
  3389. ! Fixed a SEGV issue carried with last release. Improved handling
  3390. of communication channels between core process and plugins.
  3391. ! Uniformed plugin's handling of signals; now sending a SIGINT to
  3392. all pmacctd processes causes it to flush caches and exit nicely.
  3393. ! Updated documentation; still no man page.
  3394. 0.6.1 -- 24-Mar-2004
  3395. + A new concept has been introduced: plugin names. A name could
  3396. be assigned to each running plugin allowing to run more
  3397. instances of the same plugin type; each one is configurable
  3398. with global or 'named' keys. Take a look to examples for
  3399. further info.
  3400. + Added support for PPPOE links. The code has been fully contributed
  3401. by Vasiliy Ponomarev. A big thank goes to him.
  3402. + Added a 'sql_startup_delay' configuration key to allow more
  3403. plugin instances that need to write to the DB, to flush their
  3404. data at same intervals but in different times to avoid locking
  3405. stalls or DB overkills.
  3406. + Improved handling of syslog connections. SIGHUP signal, used to
  3407. reopen a connection with syslog (eg. for log rotation purposes),
  3408. now is supported in all plugins.
  3409. + A simple LRU (Last Recently Used) cache has been added to the
  3410. in-memory table plugin. The cache gives great benefits (exploiting
  3411. some kind of locality in communication flows) when the table gets
  3412. large (and chain in buckets become long and expensive to traverse).
  3413. + Down-up of listening interface are now handled properly. Such an
  3414. event traps a reopening of connection with libpcap. [EXPERIMENTAL]
  3415. + Some work has been done (mostly via directives to preprocessor)
  3416. in order to get pmacct compiled under Solaris. [HIGLY EXPERIMENTAL,
  3417. translates: don't assume it works but, please, try it out and some
  3418. kind of feedback would be appreciated]
  3419. ! Plugins have been better structured; plugin hooking has been
  3420. simplified and re-documented; configuration parser has been
  3421. strongly improved.
  3422. ! Fixed a bug in 'configure' script; when supplying custom paths to
  3423. MySQL libraries an erroneous library filename was searched for.
  3424. (thanks to Wim Kerkhoff)
  3425. 0.6.0p3 -- 09-Feb-2004
  3426. ! Fixed an issue concerning promiscuous mode; it was
  3427. erroneously defaulting to 'false' under certain
  3428. conditions. (Thanks to Royston Boot for signalling the
  3429. problem)
  3430. 0.6.0p2 -- 05-Feb-2004
  3431. ! Fixed pmacct daemon in-memory table plugin unstability,
  3432. noticed under sustained loads. (A thank for signalling
  3433. the problem goes to Martin Pot)
  3434. ! Minor code rewritings for better optimizazion done in
  3435. both in-memory table plugin and pmacct client.
  3436. 0.6.0p1 -- 28-Jan-2004
  3437. ! Fixed a bug in in-memory table plugin that was causing
  3438. incorrect memorization of statistics. (Many thanks for
  3439. promptly signalling it go to Martin Pot)
  3440. ! Fixed a bug in pmacct client, used to gather stats from
  3441. in-memory table. Under high loads and certain conditions
  3442. the client was returning SEGV due to a realloc() issue.
  3443. (Thanks to Martin Pot)
  3444. 0.6.0 -- 27-Jan-2004
  3445. + PMACCT OPENS TO POSTGRESQL: fully featured PostgreSQL
  3446. plugin has been added; it's transaction based and
  3447. already supports "recovery mode" both via logfile and
  3448. backup DB actions. pmpgplay is the new tool that allows
  3449. to play logfiles written in recovery mode by the plugin
  3450. into a PostgreSQL DB. See CONFIG-KEYS and EXAMPLES for
  3451. further informations. (Again, many thanks to Wim Kerkoff)
  3452. + Added new "recovery mode" action to MySQL plugin: write
  3453. data to a backup DB if primary DB fails. DB table/user/
  3454. password need to be the same as in the primary DB. The
  3455. action could be enabled via "sql_backup_host" config
  3456. key.
  3457. + Added a "sql_data" configuration optinion; a "frontend"
  3458. value means to write human readable (strings) data; a
  3459. "backend" value means to write integers in network byte
  3460. order. Currently, this option is supported only into the
  3461. new PostgreSQL plugin. See CONFIG-KEYS and README.pgsql
  3462. for further informations.
  3463. + Added support for simple password authentication in
  3464. client/server query mechanism for in-memory table
  3465. statistics. It's available via "imt_passwd" config key.
  3466. + Added a "-t" commandline switch to pmmyplay; it runs
  3467. the tool in a test only mode; useful to check header
  3468. infos or logfile integrity.
  3469. ! Fixed an ugly bug that made impossible MAC accounting
  3470. over certain links. Was affected only version 0.5.4.
  3471. ! Many code and structure cleanups.
  3472. 0.5.4 -- 18-Dec-2003
  3473. + Added a commandline and configuration switch to use
  3474. or not promiscuous mode for traffic capturing; useful
  3475. to avoid waste of resources if running over a router.
  3476. + Introduced a "recovery mode" concept for MySQL plugin:
  3477. if DB fails an action is taken; currently is possible
  3478. to write data to a logfile. More failover solutions to
  3479. come in next releases. Thanks also to Wim Kerkhoff.
  3480. + Added a new "pmmyplay" tool. Allows to play logfiles
  3481. previously written by a MySQL plugin in recovery mode.
  3482. Check EXAMPLES for hints; see INTERNALS for further
  3483. details about recovery mode and pmmyplay.
  3484. + Added syslog logging and debugging. Thanks for long
  3485. brainstormings to Wim Kerkhoff.
  3486. + Added chance to write PID of pmacctd core process to
  3487. a specified file; it could help in automating tasks
  3488. that need to send signals to pmacctd (eg. to rotate
  3489. logfiles and reopen syslog connection). Take a look
  3490. to SIGNALS file for further informations.
  3491. + support for 802.11 Wireless links. [EXPERIMENTAL]
  3492. + support for linux cooked device links (DLT_LINUX_SLL).
  3493. pcap library >= 0.6.x is needed. A big thank goes to
  3494. KP Kirchdoerfer.
  3495. ! Simplified client/server query mechanism; avoided all
  3496. string comparison stuff.
  3497. ! Large parts of in-memory table plugin code has been
  3498. revised to achieve better efficiency and optimization of
  3499. available resources.
  3500. 0.5.3 -- 20-Nov-2003
  3501. ! pmacctd core has been optimized and a new loop-callback
  3502. scheme driven by pcap library has been introduced; I/O
  3503. multiplexing is avoided.
  3504. ! In MySQL plugin, refresh of entries in the DB has been
  3505. switched from a signal-driven approach to a lazy timeslot
  3506. based one. If using historical recording, taking care
  3507. to the choosen values, this greatly alleviates cache
  3508. aliasing.
  3509. ! In MySQL plugin, modulo function (for insertion of data in
  3510. the direct mapped cache) has been changed: crc32 algorithm
  3511. has been adopted. Experimental tests shown the reduction of
  3512. cache aliasing to about 0.45%.
  3513. ! The whole MySQL plugin has been inspected for performance
  3514. bottlenecks resulted by the addition of new features in
  3515. last releases.
  3516. ! Fixed a bug in link layer handlers.
  3517. 0.5.2 -- 03-Nov-2003
  3518. + "sql_history" configuration key syntax has been changed to
  3519. support history recording at fixed times with mins, hrs and
  3520. days granularity. A little of date arithmetics has been
  3521. introduced (merely multiplicative factors, eg. to ease 95th
  3522. percentile operations).
  3523. + Added "sql_history_roundoff" configuration key to round off
  3524. time of first timeslot. This little care gives cleaner time
  3525. results and inductively affects all subsequent slots.
  3526. + Achieved more precise calculations via timestamps added to
  3527. the cache structure to avoid data counted during the current
  3528. timeslot and not already fed in the DB to be accounted in next
  3529. slot.
  3530. ! Monthly historical aggregation is no more available.
  3531. ! Fixed portability issues posed by vsnprintf() in MySQL
  3532. plugin. Now the plugin compiles smoothly under Tru64 Unix.
  3533. 0.5.1 -- 01-Oct-2003
  3534. + due to the proliferation of command-line options, the
  3535. support for a configuration file has been added. All
  3536. commandline switches until version 0.5.0 will be supported
  3537. in the future.
  3538. New configurable options (eg. log to a remote SQL server)
  3539. will be only supported via configuration file. See
  3540. CONFIG-KEYS file for available configuration keys.
  3541. + added support for historical recording of counters in the
  3542. MySQL database. Available granularities of aggregation are
  3543. hourly, daily or monthly (eg. counters are separated hour
  3544. by hour, daily of monthly for each record). Timestamps of
  3545. last INSERT and UPDATE have been added over each record.
  3546. (thanks to Wim Kerkhoff for his strong collaboration)
  3547. + support for IP header options.
  3548. + support for PPP links. [EXPERIMENTAL]
  3549. ! Fixed a MySQL plugin direct-mapped cache issue: the cache
  3550. now traps INSERT queries when an UPDATE fails due to any
  3551. asyncronous table manipulation event (eg. external scripts,
  3552. table truncation, etc.).
  3553. ! MySQL plugin has been strongly revised and optimized; added
  3554. options to save data to a remote sql server and to customize
  3555. username, password and table; added MySQL locking stuff.
  3556. (another big thank to Wim Kerkhoff).
  3557. ! various code cleanups.
  3558. 0.5.0 -- 22-Jul-2003
  3559. + static aggregation directives (src_host, dst_host, ..)
  3560. are now superseded by primitives that can be stacked
  3561. together to form complex aggregation methods.
  3562. The commandline syntax of the client program has been
  3563. consequently changed to support these new features.
  3564. + two new primitives have been added: source MAC address
  3565. and destination MAC address.
  3566. + support for 802.1Q (VLANs) tagged packets (thanks to
  3567. Rich Gade).
  3568. + support for FDDI links. [EXPERIMENTAL]
  3569. ! the core pmacctd loop (that gathers packets off the
  3570. wire and feeds data to plugins) has been revised and
  3571. strongly optimized.
  3572. ! the main loop of MySQL plugin has been optimized with
  3573. the introduction of adaptive selection queries during
  3574. the update process.
  3575. ! fixed a memory allocation issue (that caused a SIGSEGV,
  3576. under certain circustances) in pmacct client: now the
  3577. upper bound of dss is checked for large data retrieval.
  3578. 0.4.2 -- 20-Jun-2003
  3579. + limited support for transport protocols (currently
  3580. only tcp and udp): aggregation of statistics for
  3581. source or destination port.
  3582. + optimized query mechanism for in-memory table; solved
  3583. few generalization issues that will enable (in future
  3584. versions) to support complex queries.
  3585. + added "-t" pmacctd commandline switch to specify a
  3586. custom database table.
  3587. ! fixed realloc() issue in pmacct client (thanks to
  3588. Arjen Nienhuis).
  3589. ! fixed an issue regarding mysql headers in the configure
  3590. script.
  3591. 0.4.1 -- 08-May-2003
  3592. ! missing break in a case statement that led pmacctd
  3593. to misbehaviours; a cleaner approach to global vars
  3594. (thanks to Peter Payne).
  3595. ! fixed an issue with getopt() and external vars. Now
  3596. pmacct has reported to compile without problems on
  3597. FreeBSD 4.x (thanks to Kirill Ponomarew).
  3598. ! missing conditional statement to check the runtime
  3599. execution of compiled plugins in exec_plugins()
  3600. 0.4.0 -- 02-May-2003
  3601. + switched to a plugin architecture: plugins need to
  3602. be activated at configure time to be compiled and
  3603. then used via "-P" command-line switch in pmacctd.
  3604. See PLUGINS for more details.
  3605. + added first plugin: Mysql driver. It uses a Mysql
  3606. database as backend to store statistics other than
  3607. in-memory table. See sql/ directory for scripts for
  3608. creation of db needed to store data.
  3609. + added the choice to collect statistics for traffic
  3610. flows in addition to src|dst|sum aggregation via
  3611. the "-c flows" command-line switch in pmacctd.
  3612. + major code cleanups.
  3613. + mostly rewritten configure script; switched back to
  3614. autoconf 2.1.
  3615. 0.3.4 -- 24-Mar-2003
  3616. + accounting of IP traffic for source, destination
  3617. and aggregation of both. Introduced -c switch to
  3618. pmacctd (thanks to Martynas Bieliauskas).
  3619. + added daemonization of pmacctd process via -D
  3620. command line switch
  3621. + added buffering via pcap_open_live() timeout handling
  3622. on those architectures where it is supported.
  3623. + It compiles and works fine over FreeBSD 5.x;
  3624. solved some pcap library issues.
  3625. + added customization of pipe for client/server
  3626. communication via -p command line switch both in
  3627. pmacct and pmacctd
  3628. 0.3.3 -- 19-Mar-2003
  3629. + introduced synchronous I/O multiplexing
  3630. + support for -m 0 pmacctd switch, in-memory table
  3631. can grow undefinitely.
  3632. + revised memory pool descriptors table structure
  3633. ! introduced realloc() in pmacct to support really
  3634. large in-memory table transfers; solved additional
  3635. alignment problems.
  3636. ! solved compatibility issues with libpcap 0.4
  3637. ! solved nasty problem with -i pmacctd switch
  3638. ! solved various memory code bugs and open issues
  3639. 0.3.2 -- 13-Mar-2003
  3640. + support for pcap library filters
  3641. ! minor bugfixes
  3642. 0.3.1 -- 12-Mar-2003
  3643. + documentation stuff: updated TODO and added INTERNALS
  3644. + revised query mechanism to server process, added a
  3645. standard header to find command and optional values
  3646. carried in query buffer.
  3647. + added -s commandline switch to customize the size of
  3648. each memory pool; see INTERNLS for more informations
  3649. ! stability tests and fixes
  3650. ! configure script enhancements
  3651. 0.3.0 -- 11-Mar-2003
  3652. ! not public release
  3653. + increased efficiency through allocation of memory pools
  3654. instead of sparse malloc() calls when inserting new
  3655. elements in in-memory table.
  3656. + added -m commandline switch to pmacctd to set the number
  3657. of available memory pools; the size of each memory pool is
  3658. the number of buckets, chosen with -b commandline option,
  3659. see INTERNALS for more informations.
  3660. + switched client program to getopt() to acquire commandline
  3661. inputs.
  3662. + new -m commandline option in client program to acquire
  3663. statistics of a specified IP address in a format useful for
  3664. acquisition by MRTG program; see examples directory for a
  3665. sample mrtg configuration.
  3666. ! major bugfixes
  3667. ! minor code cleanups
  3668. 0.2.4 -- 07-Mar-2003
  3669. + portability: Tru64 5.x
  3670. ! configure script fixes
  3671. ! minor bugfixes
  3672. 0.2.3 -- 05-Mar-2003
  3673. + first public release
  3674. ! portability fixes
  3675. ! minor bugfixes
  3676. 0.2.2 -- 04-Mar-2003
  3677. + minor code cleanups
  3678. + added autoconf, automake stuff
  3679. 0.2.1 -- 03-Mar-2003
  3680. + fork()ing when handling queries
  3681. + signal handling
  3682. + command-line options using getopt()
  3683. + usage instructions
  3684. ! major bugfixes
  3685. 0.2.0 -- 01-Mar-2003
  3686. + dynamic allocation of in-memory table
  3687. + query (client/server) mechanism
  3688. + added a Makefile
  3689. ! major bugfixes
  3690. 0.1.0 -- late Feb, 2003
  3691. + Initial release