delete duplicates items from a list
Question:
i have the following list
list1=['philips, be, currently, monitor, development, and, update, relate, to, the, recently, release, oracle, security, update, concern, a, vulnerability, within, the, e, business, suite, product, oracle, have, release, a, patch, and, recommend, that, it, be, apply, as, soon, as, possibleat, this, point, of, time, no, philips, product, be, know, to, be, impact, in, accordance, with, philips, global, security, policy, philips, continue, to, analyze, the, matter, and, further, information, will, be, post, on, the, philips, product, security, advisory, page, as, appropriatephilip, be, commit, to, ensure, the, safety, security, integrity, and, regulatory, compliance, of, our, product, to, be, deploy, and, to, operate, within, philips, approve, product, specification, therefore, in, accordance, with, philips, policy, and, regulatory, requirement, all, change, of, configuration, or, software, to, philips, product, may, be, implement, only, in, accordance, with, philips, product, specific, verify, validate, authorized, and, communicate, customer, procedure, or, field, actionsif, a, product, do, require, operate, system, security, update, configuration, change, or, other, action, to, be, take, by, our, customer, or, by, philips, customer, services, product, specific, service, documentation, will, be, produce, by, philips, product, team, and, make, available, to, philips, service, delivery, platform, such, as, the, philips, incenter, customer, portalcontract, entitle, customer, may, use, philips, incenter, and, be, encourage, to, request, philips, incenter, access, and, reference, product, specific, information, post, if, customer, still, have, question, all, customer, be, encourage, to, contact, their, local, service, support, team, or, regional, product, service, support, as, appropriate, for, up, to, date, information, specific, to, their, philips, product',
'philips, be, currently, monitor, development, and, update, relate, to, the, recently, release, ciscoadvisory, cisco, have, confirm, that, a, critical, vulnerability, exist, in, their, adaptive, security, appliance, software, and, firepower, threat, defense, softwaresuccessful, execution, of, this, vulnerability, could, allow, an, unauthenticated, remote, attacker, to, because, a, denial, of, service, condition, on, an, affected, device, cisco, have, release, software, update, that, help, remediate, this, vulnerabilityat, this, time, no, philips, product, be, know, to, be, impact, in, accordance, with, philips, global, security, policy, philips, continue, to, analyze, the, matter, and, further, information, will, be, post, on, the, philips, product, security, advisory, page, as, appropriatephilip, be, commit, to, ensure, the, safety, security, integrity, and, regulatory, compliance, of, our, product, to, be, deploy, and, to, operate, within, philips, approve, product, specification, therefore, in, accordance, with, philipss, policy, and, regulatory, requirement, all, change, of, configuration, or, software, to, philips, product, may, be, implement, only, in, accordance, with, philipss, product, specific, verify, validate, authorized, and, communicate, customer, procedure, or, field, actionsif, a, product, do, require, operate, system, security, update, configuration, change, or, other, action, to, be, take, by, our, customer, or, by, philips, customer, services, product, specific, service, documentation, will, be, produce, by, philipss, product, team, and, make, available, to, philips, service, delivery, platform, such, as, the, philips, incenter, customer, portalcontract, entitle, customer, may, use, philips, incenter, and, be, encourage, to, request, philips, incenter, access, and, reference, product, specific, information, post, if, customer, still, have, question, all, customer, be, encourage, to, contact, their, local, service, support, team, or, regional, product, service, support, as, appropriate, for, up, to, date, information, specific, to, their, philips, product']
I am trying to delete the duplicates:
mylist = sorted(set(list1))
mylist
['philips, be, currently, monitor, development, and, update, relate, to, the, recently, release, ciscoadvisory, cisco, have, confirm, that, a, critical, vulnerability, exist, in, their, adaptive, security, appliance, software, and, firepower, threat, defense, softwaresuccessful, execution, of, this, vulnerability, could, allow, an, unauthenticated, remote, attacker, to, because, a, denial, of, service, condition, on, an, affected, device, cisco, have, release, software, update, that, help, remediate, this, vulnerabilityat, this, time, no, philips, product, be, know, to, be, impact, in, accordance, with, philips, global, security, policy, philips, continue, to, analyze, the, matter, and, further, information, will, be, post, on, the, philips, product, security, advisory, page, as, appropriatephilip, be, commit, to, ensure, the, safety, security, integrity, and, regulatory, compliance, of, our, product, to, be, deploy, and, to, operate, within, philips, approve, product, specification, therefore, in, accordance, with, philipss, policy, and, regulatory, requirement, all, change, of, configuration, or, software, to, philips, product, may, be, implement, only, in, accordance, with, philipss, product, specific, verify, validate, authorized, and, communicate, customer, procedure, or, field, actionsif, a, product, do, require, operate, system, security, update, configuration, change, or, other, action, to, be, take, by, our, customer, or, by, philips, customer, services, product, specific, service, documentation, will, be, produce, by, philipss, product, team, and, make, available, to, philips, service, delivery, platform, such, as, the, philips, incenter, customer, portalcontract, entitle, customer, may, use, philips, incenter, and, be, encourage, to, request, philips, incenter, access, and, reference, product, specific, information, post, if, customer, still, have, question, all, customer, be, encourage, to, contact, their, local, service, support, team, or, regional, product, service, support, as, appropriate, for, up, to, date, information, specific, to, their, philips, product',
'philips, be, currently, monitor, development, and, update, relate, to, the, recently, release, oracle, security, update, concern, a, vulnerability, within, the, e, business, suite, product, oracle, have, release, a, patch, and, recommend, that, it, be, apply, as, soon, as, possibleat, this, point, of, time, no, philips, product, be, know, to, be, impact, in, accordance, with, philips, global, security, policy, philips, continue, to, analyze, the, matter, and, further, information, will, be, post, on, the, philips, product, security, advisory, page, as, appropriatephilip, be, commit, to, ensure, the, safety, security, integrity, and, regulatory, compliance, of, our, product, to, be, deploy, and, to, operate, within, philips, approve, product, specification, therefore, in, accordance, with, philips, policy, and, regulatory, requirement, all, change, of, configuration, or, software, to, philips, product, may, be, implement, only, in, accordance, with, philips, product, specific, verify, validate, authorized, and, communicate, customer, procedure, or, field, actionsif, a, product, do, require, operate, system, security, update, configuration, change, or, other, action, to, be, take, by, our, customer, or, by, philips, customer, services, product, specific, service, documentation, will, be, produce, by, philips, product, team, and, make, available, to, philips, service, delivery, platform, such, as, the, philips, incenter, customer, portalcontract, entitle, customer, may, use, philips, incenter, and, be, encourage, to, request, philips, incenter, access, and, reference, product, specific, information, post, if, customer, still, have, question, all, customer, be, encourage, to, contact, their, local, service, support, team, or, regional, product, service, support, as, appropriate, for, up, to, date, information, specific, to, their, philips, product']
As you can see this does not happen and the initial is still preserved.
Any idea why this happens?
Answers:
Your problem is that the list1 contains just one element. Creating a set will not have any obvious effect because there are no duplicates. I think this is what you want to achieve:
list(set(list1[0].split(', ')))
If the list has more than one element, you may want to do this:
s = set()
for e in list1:
for word in e.split(', '):
s.add(word)
print(list(s))
Both of the strings in your array are not equal.
If you want to remove duplicates from inside the strings, you must split first.
Try doing:
list1 = [
'philips, be, currently, monitor, development, and, update, relate, to, the, recently, release, oracle, security, '
'update, concern, a, vulnerability, within, the, e, business, suite, product, oracle, have, release, a, patch, '
'and, recommend, that, it, be, apply, as, soon, as, possibleat, this, point, of, time, no, philips, product, be, '
'know, to, be, impact, in, accordance, with, philips, global, security, policy, philips, continue, to, analyze, '
'the, matter, and, further, information, will, be, post, on, the, philips, product, security, advisory, page, as, '
'appropriatephilip, be, commit, to, ensure, the, safety, security, integrity, and, regulatory, compliance, of, '
'our, product, to, be, deploy, and, to, operate, within, philips, approve, product, specification, therefore, in, '
'accordance, with, philips, policy, and, regulatory, requirement, all, change, of, configuration, or, software, '
'to, philips, product, may, be, implement, only, in, accordance, with, philips, product, specific, verify, '
'validate, authorized, and, communicate, customer, procedure, or, field, actionsif, a, product, do, require, '
'operate, system, security, update, configuration, change, or, other, action, to, be, take, by, our, customer, '
'or, by, philips, customer, services, product, specific, service, documentation, will, be, produce, by, philips, '
'product, team, and, make, available, to, philips, service, delivery, platform, such, as, the, philips, incenter, '
'customer, portalcontract, entitle, customer, may, use, philips, incenter, and, be, encourage, to, request, '
'philips, incenter, access, and, reference, product, specific, information, post, if, customer, still, have, '
'question, all, customer, be, encourage, to, contact, their, local, service, support, team, or, regional, '
'product, service, support, as, appropriate, for, up, to, date, information, specific, to, their, philips, '
'product',
'philips, be, currently, monitor, development, and, update, relate, to, the, recently, release, ciscoadvisory, '
'cisco, have, confirm, that, a, critical, vulnerability, exist, in, their, adaptive, security, appliance, '
'software, and, firepower, threat, defense, softwaresuccessful, execution, of, this, vulnerability, could, allow, '
'an, unauthenticated, remote, attacker, to, because, a, denial, of, service, condition, on, an, affected, device, '
'cisco, have, release, software, update, that, help, remediate, this, vulnerabilityat, this, time, no, philips, '
'product, be, know, to, be, impact, in, accordance, with, philips, global, security, policy, philips, continue, '
'to, analyze, the, matter, and, further, information, will, be, post, on, the, philips, product, security, '
'advisory, page, as, appropriatephilip, be, commit, to, ensure, the, safety, security, integrity, and, '
'regulatory, compliance, of, our, product, to, be, deploy, and, to, operate, within, philips, approve, product, '
'specification, therefore, in, accordance, with, philipss, policy, and, regulatory, requirement, all, change, of, '
'configuration, or, software, to, philips, product, may, be, implement, only, in, accordance, with, philipss, '
'product, specific, verify, validate, authorized, and, communicate, customer, procedure, or, field, actionsif, a, '
'product, do, require, operate, system, security, update, configuration, change, or, other, action, to, be, take, '
'by, our, customer, or, by, philips, customer, services, product, specific, service, documentation, will, be, '
'produce, by, philipss, product, team, and, make, available, to, philips, service, delivery, platform, such, as, '
'the, philips, incenter, customer, portalcontract, entitle, customer, may, use, philips, incenter, and, be, '
'encourage, to, request, philips, incenter, access, and, reference, product, specific, information, post, if, '
'customer, still, have, question, all, customer, be, encourage, to, contact, their, local, service, support, '
'team, or, regional, product, service, support, as, appropriate, for, up, to, date, information, specific, to, '
'their, philips, product']
list2 = []
for i in list1:
list2 += i.split(', ')
print("Length of list before removing duplicated =", len(list2))
print("Length of list before after duplicated =", len(set(list2)))
Output is:
filtered_list = list({ word for phrase in list1 for word in phrase.split(', ') })
i have the following list
list1=['philips, be, currently, monitor, development, and, update, relate, to, the, recently, release, oracle, security, update, concern, a, vulnerability, within, the, e, business, suite, product, oracle, have, release, a, patch, and, recommend, that, it, be, apply, as, soon, as, possibleat, this, point, of, time, no, philips, product, be, know, to, be, impact, in, accordance, with, philips, global, security, policy, philips, continue, to, analyze, the, matter, and, further, information, will, be, post, on, the, philips, product, security, advisory, page, as, appropriatephilip, be, commit, to, ensure, the, safety, security, integrity, and, regulatory, compliance, of, our, product, to, be, deploy, and, to, operate, within, philips, approve, product, specification, therefore, in, accordance, with, philips, policy, and, regulatory, requirement, all, change, of, configuration, or, software, to, philips, product, may, be, implement, only, in, accordance, with, philips, product, specific, verify, validate, authorized, and, communicate, customer, procedure, or, field, actionsif, a, product, do, require, operate, system, security, update, configuration, change, or, other, action, to, be, take, by, our, customer, or, by, philips, customer, services, product, specific, service, documentation, will, be, produce, by, philips, product, team, and, make, available, to, philips, service, delivery, platform, such, as, the, philips, incenter, customer, portalcontract, entitle, customer, may, use, philips, incenter, and, be, encourage, to, request, philips, incenter, access, and, reference, product, specific, information, post, if, customer, still, have, question, all, customer, be, encourage, to, contact, their, local, service, support, team, or, regional, product, service, support, as, appropriate, for, up, to, date, information, specific, to, their, philips, product',
'philips, be, currently, monitor, development, and, update, relate, to, the, recently, release, ciscoadvisory, cisco, have, confirm, that, a, critical, vulnerability, exist, in, their, adaptive, security, appliance, software, and, firepower, threat, defense, softwaresuccessful, execution, of, this, vulnerability, could, allow, an, unauthenticated, remote, attacker, to, because, a, denial, of, service, condition, on, an, affected, device, cisco, have, release, software, update, that, help, remediate, this, vulnerabilityat, this, time, no, philips, product, be, know, to, be, impact, in, accordance, with, philips, global, security, policy, philips, continue, to, analyze, the, matter, and, further, information, will, be, post, on, the, philips, product, security, advisory, page, as, appropriatephilip, be, commit, to, ensure, the, safety, security, integrity, and, regulatory, compliance, of, our, product, to, be, deploy, and, to, operate, within, philips, approve, product, specification, therefore, in, accordance, with, philipss, policy, and, regulatory, requirement, all, change, of, configuration, or, software, to, philips, product, may, be, implement, only, in, accordance, with, philipss, product, specific, verify, validate, authorized, and, communicate, customer, procedure, or, field, actionsif, a, product, do, require, operate, system, security, update, configuration, change, or, other, action, to, be, take, by, our, customer, or, by, philips, customer, services, product, specific, service, documentation, will, be, produce, by, philipss, product, team, and, make, available, to, philips, service, delivery, platform, such, as, the, philips, incenter, customer, portalcontract, entitle, customer, may, use, philips, incenter, and, be, encourage, to, request, philips, incenter, access, and, reference, product, specific, information, post, if, customer, still, have, question, all, customer, be, encourage, to, contact, their, local, service, support, team, or, regional, product, service, support, as, appropriate, for, up, to, date, information, specific, to, their, philips, product']
I am trying to delete the duplicates:
mylist = sorted(set(list1))
mylist
['philips, be, currently, monitor, development, and, update, relate, to, the, recently, release, ciscoadvisory, cisco, have, confirm, that, a, critical, vulnerability, exist, in, their, adaptive, security, appliance, software, and, firepower, threat, defense, softwaresuccessful, execution, of, this, vulnerability, could, allow, an, unauthenticated, remote, attacker, to, because, a, denial, of, service, condition, on, an, affected, device, cisco, have, release, software, update, that, help, remediate, this, vulnerabilityat, this, time, no, philips, product, be, know, to, be, impact, in, accordance, with, philips, global, security, policy, philips, continue, to, analyze, the, matter, and, further, information, will, be, post, on, the, philips, product, security, advisory, page, as, appropriatephilip, be, commit, to, ensure, the, safety, security, integrity, and, regulatory, compliance, of, our, product, to, be, deploy, and, to, operate, within, philips, approve, product, specification, therefore, in, accordance, with, philipss, policy, and, regulatory, requirement, all, change, of, configuration, or, software, to, philips, product, may, be, implement, only, in, accordance, with, philipss, product, specific, verify, validate, authorized, and, communicate, customer, procedure, or, field, actionsif, a, product, do, require, operate, system, security, update, configuration, change, or, other, action, to, be, take, by, our, customer, or, by, philips, customer, services, product, specific, service, documentation, will, be, produce, by, philipss, product, team, and, make, available, to, philips, service, delivery, platform, such, as, the, philips, incenter, customer, portalcontract, entitle, customer, may, use, philips, incenter, and, be, encourage, to, request, philips, incenter, access, and, reference, product, specific, information, post, if, customer, still, have, question, all, customer, be, encourage, to, contact, their, local, service, support, team, or, regional, product, service, support, as, appropriate, for, up, to, date, information, specific, to, their, philips, product',
'philips, be, currently, monitor, development, and, update, relate, to, the, recently, release, oracle, security, update, concern, a, vulnerability, within, the, e, business, suite, product, oracle, have, release, a, patch, and, recommend, that, it, be, apply, as, soon, as, possibleat, this, point, of, time, no, philips, product, be, know, to, be, impact, in, accordance, with, philips, global, security, policy, philips, continue, to, analyze, the, matter, and, further, information, will, be, post, on, the, philips, product, security, advisory, page, as, appropriatephilip, be, commit, to, ensure, the, safety, security, integrity, and, regulatory, compliance, of, our, product, to, be, deploy, and, to, operate, within, philips, approve, product, specification, therefore, in, accordance, with, philips, policy, and, regulatory, requirement, all, change, of, configuration, or, software, to, philips, product, may, be, implement, only, in, accordance, with, philips, product, specific, verify, validate, authorized, and, communicate, customer, procedure, or, field, actionsif, a, product, do, require, operate, system, security, update, configuration, change, or, other, action, to, be, take, by, our, customer, or, by, philips, customer, services, product, specific, service, documentation, will, be, produce, by, philips, product, team, and, make, available, to, philips, service, delivery, platform, such, as, the, philips, incenter, customer, portalcontract, entitle, customer, may, use, philips, incenter, and, be, encourage, to, request, philips, incenter, access, and, reference, product, specific, information, post, if, customer, still, have, question, all, customer, be, encourage, to, contact, their, local, service, support, team, or, regional, product, service, support, as, appropriate, for, up, to, date, information, specific, to, their, philips, product']
As you can see this does not happen and the initial is still preserved.
Any idea why this happens?
Your problem is that the list1 contains just one element. Creating a set will not have any obvious effect because there are no duplicates. I think this is what you want to achieve:
list(set(list1[0].split(', ')))
If the list has more than one element, you may want to do this:
s = set()
for e in list1:
for word in e.split(', '):
s.add(word)
print(list(s))
Both of the strings in your array are not equal.
If you want to remove duplicates from inside the strings, you must split first.
Try doing:
list1 = [
'philips, be, currently, monitor, development, and, update, relate, to, the, recently, release, oracle, security, '
'update, concern, a, vulnerability, within, the, e, business, suite, product, oracle, have, release, a, patch, '
'and, recommend, that, it, be, apply, as, soon, as, possibleat, this, point, of, time, no, philips, product, be, '
'know, to, be, impact, in, accordance, with, philips, global, security, policy, philips, continue, to, analyze, '
'the, matter, and, further, information, will, be, post, on, the, philips, product, security, advisory, page, as, '
'appropriatephilip, be, commit, to, ensure, the, safety, security, integrity, and, regulatory, compliance, of, '
'our, product, to, be, deploy, and, to, operate, within, philips, approve, product, specification, therefore, in, '
'accordance, with, philips, policy, and, regulatory, requirement, all, change, of, configuration, or, software, '
'to, philips, product, may, be, implement, only, in, accordance, with, philips, product, specific, verify, '
'validate, authorized, and, communicate, customer, procedure, or, field, actionsif, a, product, do, require, '
'operate, system, security, update, configuration, change, or, other, action, to, be, take, by, our, customer, '
'or, by, philips, customer, services, product, specific, service, documentation, will, be, produce, by, philips, '
'product, team, and, make, available, to, philips, service, delivery, platform, such, as, the, philips, incenter, '
'customer, portalcontract, entitle, customer, may, use, philips, incenter, and, be, encourage, to, request, '
'philips, incenter, access, and, reference, product, specific, information, post, if, customer, still, have, '
'question, all, customer, be, encourage, to, contact, their, local, service, support, team, or, regional, '
'product, service, support, as, appropriate, for, up, to, date, information, specific, to, their, philips, '
'product',
'philips, be, currently, monitor, development, and, update, relate, to, the, recently, release, ciscoadvisory, '
'cisco, have, confirm, that, a, critical, vulnerability, exist, in, their, adaptive, security, appliance, '
'software, and, firepower, threat, defense, softwaresuccessful, execution, of, this, vulnerability, could, allow, '
'an, unauthenticated, remote, attacker, to, because, a, denial, of, service, condition, on, an, affected, device, '
'cisco, have, release, software, update, that, help, remediate, this, vulnerabilityat, this, time, no, philips, '
'product, be, know, to, be, impact, in, accordance, with, philips, global, security, policy, philips, continue, '
'to, analyze, the, matter, and, further, information, will, be, post, on, the, philips, product, security, '
'advisory, page, as, appropriatephilip, be, commit, to, ensure, the, safety, security, integrity, and, '
'regulatory, compliance, of, our, product, to, be, deploy, and, to, operate, within, philips, approve, product, '
'specification, therefore, in, accordance, with, philipss, policy, and, regulatory, requirement, all, change, of, '
'configuration, or, software, to, philips, product, may, be, implement, only, in, accordance, with, philipss, '
'product, specific, verify, validate, authorized, and, communicate, customer, procedure, or, field, actionsif, a, '
'product, do, require, operate, system, security, update, configuration, change, or, other, action, to, be, take, '
'by, our, customer, or, by, philips, customer, services, product, specific, service, documentation, will, be, '
'produce, by, philipss, product, team, and, make, available, to, philips, service, delivery, platform, such, as, '
'the, philips, incenter, customer, portalcontract, entitle, customer, may, use, philips, incenter, and, be, '
'encourage, to, request, philips, incenter, access, and, reference, product, specific, information, post, if, '
'customer, still, have, question, all, customer, be, encourage, to, contact, their, local, service, support, '
'team, or, regional, product, service, support, as, appropriate, for, up, to, date, information, specific, to, '
'their, philips, product']
list2 = []
for i in list1:
list2 += i.split(', ')
print("Length of list before removing duplicated =", len(list2))
print("Length of list before after duplicated =", len(set(list2)))
Output is:
filtered_list = list({ word for phrase in list1 for word in phrase.split(', ') })