Fix various memory leaks when receiving packets #1847

gpotter2 · 2019-02-14T20:37:51Z

This PR:

removes threaded mode: it was a pretty poor idea. What really makes the difference is the use of generators, not the multi threading on sending
unify docs
fix randomSixLoWPAN bug
fixes Scapy 2.4.2 uses too much memory when sending packets #1897 => close all created pcap objects, to avoid a memory overflow
fixes stop_filter is causing memory leaks #1918 => correctly close sockets when count or stop_filter conditions are triggered in sniff

codecov · 2019-02-14T23:08:04Z

Codecov Report

Merging #1847 into master will decrease coverage by 2.02%.
The diff coverage is 54.83%.

@@            Coverage Diff             @@
##           master    #1847      +/-   ##
==========================================
- Coverage   85.75%   83.72%   -2.03%     
==========================================
  Files         185      182       -3     
  Lines       42476    41546     -930     
==========================================
- Hits        36424    34785    -1639     
- Misses       6052     6761     +709

Impacted Files	Coverage Δ
scapy/sendrecv.py	`62.15% <54.83%> (-23.22%)`	⬇️
scapy/contrib/cansocket_python_can.py	`37.7% <0%> (-54.1%)`	⬇️
scapy/contrib/cansocket_native.py	`29.57% <0%> (-50.71%)`	⬇️
scapy/modules/nmap.py	`61.94% <0%> (-35.08%)`	⬇️
scapy/arch/pcapdnet.py	`27.61% <0%> (-30.48%)`	⬇️
scapy/layers/usb.py	`51.64% <0%> (-29.68%)`	⬇️
scapy/arch/common.py	`68.47% <0%> (-22.83%)`	⬇️
scapy/consts.py	`70.83% <0%> (-20.84%)`	⬇️
scapy/supersocket.py	`53.51% <0%> (-19.15%)`	⬇️
... and 27 more

codecov · 2019-02-14T23:08:04Z

Codecov Report

Merging #1847 into master will decrease coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1847      +/-   ##
==========================================
- Coverage   85.93%   85.91%   -0.02%     
==========================================
  Files         187      187              
  Lines       42744    42741       -3     
==========================================
- Hits        36732    36722      -10     
- Misses       6012     6019       +7

Impacted Files	Coverage Δ
scapy/arch/pcapdnet.py	`57.03% <100%> (-0.69%)`	⬇️
scapy/layers/sixlowpan.py	`73.27% <100%> (+0.05%)`	⬆️
scapy/sendrecv.py	`85.25% <100%> (+0.29%)`	⬆️
scapy/layers/tls/handshake_sslv2.py	`92.45% <0%> (-1.14%)`	⬇️
scapy/layers/tls/basefields.py	`79.86% <0%> (-0.68%)`	⬇️
scapy/contrib/isotp.py	`88.31% <0%> (-0.46%)`	⬇️
scapy/layers/inet6.py	`88.18% <0%> (-0.06%)`	⬇️

scapy/packet.py

gpotter2 · 2019-02-17T11:51:30Z

Ready to be reviewed

scapy/sendrecv.py

gpotter2 · 2019-03-24T15:11:40Z

scapy/arch/pcapdnet.py

-            self.ins = open_pcap(iface, MTU, self.promisc, 100, monitor=monitor)  # noqa: E501
-            # We need to have a different interface open because of an
-            # access violation in Npcap that occurs in multi-threading
-            # (see https://github.com/nmap/nmap/issues/982)


That was fixed

gpotter2 · 2019-03-24T15:11:56Z

scapy/arch/pcapdnet.py

-                    self.ins.close()
-                if out and out != ins:
-                    self.outs.close()
-            self.closed = True


Duplication with supersocket.py

guedou · 2019-03-24T18:37:37Z

scapy/sendrecv.py

        if started_callback:
            started_callback()
-        while sniff_sockets:
+        continue_sniff = True


Is that the only relevant fix? =)

That’s for #1918

There’s a lot of unrelevant cleaning happening 😄 sorry for the messy PRs

guedou · 2019-03-24T18:40:54Z

I made a mistake and pressed enter while the merged button was selected instead of comment =/

guedou · 2019-03-24T18:41:05Z

@p-l- do you agree on this PR?

p-l- · 2019-03-24T18:42:42Z

@guedou no worries, it looks like you have approved this PR and I merged it 2 minutes after.

gpotter2 · 2019-03-24T18:43:05Z

@guedou I think he merged it 😄

guedou · 2019-03-24T18:46:54Z

Latency results with Python 3.5:
Before: ~0.005
After: ~0.025

gpotter2 · 2019-03-24T18:49:08Z

I should have splited the PRs :(

gpotter2 · 2019-03-24T19:00:30Z

I get very similar results before and after :/
Before: ~ 0.030
After: ~ 0.030
With on both a slow increase the more packets have been sent (I'm testing using latency_router.py)

Edit: but I do see a massive gap between 2.4.0 and master

gpotter2 · 2019-03-27T13:53:02Z

Could one of you look into this ? I fail to see how the performance has decreased that much since 2.4.0 l, and what is causing it :/

guedou · 2019-03-29T08:20:35Z

I will test it again, but from my first test your PR reduces the latency.

gpotter2 mentioned this pull request Feb 14, 2019

Higher latency calculation (using time and sent_time) #1791

Closed

gpotter2 force-pushed the fixlatency branch from b4c2182 to 06cf542 Compare February 14, 2019 23:07

gpotter2 added bug cleanup Performs some code clean-up labels Feb 14, 2019

gpotter2 force-pushed the fixlatency branch from 06cf542 to dd30962 Compare February 14, 2019 23:44

p-l- reviewed Feb 15, 2019

View reviewed changes

scapy/packet.py Outdated Show resolved Hide resolved

gpotter2 force-pushed the fixlatency branch 3 times, most recently from bae941e to 651eced Compare February 15, 2019 22:31

guedou requested changes Feb 18, 2019

View reviewed changes

scapy/sendrecv.py Outdated Show resolved Hide resolved

gpotter2 changed the title ~~Fix latency scapy 2.4.2~~ Improve latency tests scapy 2.4.2 Feb 18, 2019

gpotter2 mentioned this pull request Feb 22, 2019

Scapy v2.4.3 #1866

Closed

27 tasks

gpotter2 added 3 commits March 24, 2019 15:50

Fix latency scapy 2.4.2

53afe84

Improve benchmarking

d772065

Fix memory overflow

e2092ae

gpotter2 force-pushed the fixlatency branch from 651eced to e2092ae Compare March 24, 2019 15:11

gpotter2 commented Mar 24, 2019

View reviewed changes

gpotter2 mentioned this pull request Mar 24, 2019

Scapy 2.4.2 uses too much memory when sending packets #1897

Closed

fix stop_filter memory leak

186aa17

gpotter2 mentioned this pull request Mar 24, 2019

stop_filter is causing memory leaks #1918

Closed

guedou reviewed Mar 24, 2019

View reviewed changes

guedou approved these changes Mar 24, 2019

View reviewed changes

p-l- merged commit 5311c5e into secdev:master Mar 24, 2019

gpotter2 changed the title ~~Improve latency tests scapy 2.4.2~~ Fix various memory leaks when receiving packets Mar 27, 2019

gpotter2 deleted the fixlatency branch July 19, 2020 09:19

Uh oh!

Fix various memory leaks when receiving packets #1847

Fix various memory leaks when receiving packets #1847

Uh oh!

Conversation

gpotter2 commented Feb 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Feb 14, 2019

Codecov Report

Uh oh!

codecov bot commented Feb 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

gpotter2 commented Feb 17, 2019

Uh oh!

Uh oh!

gpotter2 Mar 24, 2019

Choose a reason for hiding this comment

Uh oh!

gpotter2 Mar 24, 2019

Choose a reason for hiding this comment

Uh oh!

guedou Mar 24, 2019

Choose a reason for hiding this comment

Uh oh!

gpotter2 Mar 24, 2019

Choose a reason for hiding this comment

Uh oh!

guedou commented Mar 24, 2019

Uh oh!

guedou commented Mar 24, 2019

Uh oh!

p-l- commented Mar 24, 2019

Uh oh!

gpotter2 commented Mar 24, 2019

Uh oh!

guedou commented Mar 24, 2019

Uh oh!

gpotter2 commented Mar 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gpotter2 commented Mar 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gpotter2 commented Mar 27, 2019

Uh oh!

guedou commented Mar 29, 2019 via email

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gpotter2 commented Feb 14, 2019 •

edited

Loading

codecov bot commented Feb 14, 2019 •

edited

Loading

gpotter2 commented Mar 24, 2019 •

edited

Loading

gpotter2 commented Mar 24, 2019 •

edited

Loading